feat(glm5next): support pipeline parallelism via mHC multi-stream hand-off - #51
Merged
Merged
Conversation
…d-off GLM-5.3-Flash (Glm5Next) gated PP off: no make_empty_intermediate_tensors, and the non-last-rank branch carried [hidden_states, residual] while dropping the deferred mHC hc_post state (post/comb), which the receiving rank had no way to reconstruct. This implements the boundary the way DeepSeek-V4 does it with hc_mult: the sending rank materializes its last mHC layer's deferred hc_post (one mhc_post kernel per boundary; the math is unchanged, since that is exactly the post the next rank's first layer would have folded into its fused_post_pre), so the FULL multi-stream state (tokens, n, hidden) crosses in a single "hidden_states" key. The receiving rank's first mHC layer runs standalone hc_pre on the materialized stream state; residual is re-derived there, so no extra key is transported. Verified on 4x sm80 64GB (PCIe, no P2P) with GLM-5.3-Flash AWQ W4A16: PP4 14,12,12,7 + MTP x5 boots and serves; correctness battery 9/9 at temperature 0; decode 50-93 tok/s single-stream; KV pool 1,437,333 tokens. Without PP this model cannot run on 4x64GB at all (TP4 prefill is ~6.6x slower on this PCIe topology). Signed-off-by: kk <kk@cmp170hx>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GLM-5.3-Flash (
Glm5Next) gates PP off:make_empty_intermediate_tensorsis deliberately not implemented, and the non-last-rank branch ofGlm5NextModel.forwardreturns[hidden_states, residual]while dropping the deferred mHC hc_post state (post/comb) — the receiving rank cannot reconstruct the multi-stream state, so a naive PP boot is numerically broken anyway.On 4×64 GB sm_80 (PCIe, no P2P) this means GLM-5.3-Flash cannot run at all: TP4 prefill is ~6.6× slower than PP4 on this topology.
Fix — the DSV4
hc_multhand-off patternhc_postat the boundary (exactly the post the next rank's first layer would have folded into itsfused_post_pre— math unchanged, one extra post kernel per boundary). The full multi-stream state(tokens, n, hidden)crosses in a singlehidden_stateskey.make_empty_intermediate_tensorsallocates the(B, n, hidden)stream state; the stage's first mHC layer runs standalonehc_preon it and re-derivesresidual, so no extra key is transported.Glm5NextForCausalLMand the conditional-generation wrapper alias it; the "PP gated off" comments are replaced accordingly.Verification (4× CMP 170HX, sm_80, 64 GB, PCIe no-P2P)
GLM-5.3-Flash AWQ W4A16,
PP4 14,12,12,7+ MTP×5 (needs the draft-embedding fix from #50):max_model_len=524288Without this patch the same box cannot host the model; with it, PP4 is the only viable parallelism on this hardware class.