Skip to content

dflash: pass missing NVFP4 scales to attention operations - #28000

Merged
ggerganov merged 1 commit into
ggml-org:masterfrom
JamePeng:fix-missing-dflash-nvfp4-scales
Aug 30, 2026
Merged

dflash: pass missing NVFP4 scales to attention operations#28000
ggerganov merged 1 commit into
ggml-org:masterfrom
JamePeng:fix-missing-dflash-nvfp4-scales

Conversation

@JamePeng

@JamePeng JamePeng commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

I tested the DFlash2 draft model provided by the https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal project. I found that the DFlash2 BF16 and Q8 versions performed normally, but the DFlash2 NVFP4 draft models produced almost no accepted speculative tokens because the Q, K, V, and output projection scales were not passed to the corresponding graph operations. I eventually traced the issue to a missing scale parameter pass in dflash.cpp; adding the missing code allowed it to compile and run correctly.

Overview

Pass the missing NVFP4 weight scales to the DFlash graph operations:

  • K and V projections in KV injection
  • Q, K, and V projections in decoder attention
  • Output projections in both attention paths

Without these scales, the NVFP4 draft model generated incorrect draft tokens, resulting in a near-zero acceptance rate and slower speculative decoding.

Additional information

Benchmark configuration:

  • GPU: NVIDIA GeForce RTX 5090
  • Target model: Qwen3.8-27B-Q5_K_M
  • Draft model: Qwen3.8-27b-DFlash2-NVFP4
  • Full GPU offload and flash attention
  • Greedy sampling with a fixed seed
  • Maximum draft length: 7 tokens
  • Single run with a fixed prompt
Generated tokens Master PR Before acceptance After Patch After acceptance DFlash improvement
128 34.4 t/s 0.12% 93.1 t/s 36.33% 2.71x
512 35.7 t/s 0.17% 97.6 t/s 25.59% 2.73x
1024 38.4 t/s 0.39% 119.0 t/s 33.10% 3.10x

Compared with ordinary decoding from the patched build, DFlash achieved speedups of 1.65x, 1.62x, and 1.93x for 128, 512, and 1024 generated tokens, respectively.

Requirements

Logs

Master PR(10689)
master pr.txt

Patched PR
after_patch.txt

@JamePeng
dflash: pass missing NVFP4 scales to attention operations
ef7db22 
- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
@JamePeng
JamePeng requested a review from CISC as a code owner August 30, 2026 05:37
@github-actions github-actions Bot added the model Model specific label Aug 30, 2026
@CISC CISC added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 30, 2026
@ggerganov
ggerganov merged commit cc231cb into ggml-org:master Aug 30, 2026
24 of 27 checks passed
jbooth pushed a commit to jbooth/llama.cpp that referenced this pull request Aug 30, 2026
@JamePeng @jbooth
dflash: pass missing NVFP4 scales to attention operations (ggml-org#2…
274c850 
…8000)

- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
fewtarius added a commit to fewtarius/CachyLLama that referenced this pull request Sep 1, 2026
@fewtarius
merge upstream/master (a14dba6..d086dbb): 76 commits
459bcd9 
Properly merge upstream commits preserving all CachyLLama features:
- CachyLLama Lightning Indexer shader + DSV4_HC pre/comb/post (kept over
  upstream cb30059's li_dict variant - c22b79e already removed the dup)
- FA dequant-once scratch with host-RAM safety gate (q8_0/q4_0/q4_1/q5_0/q5_1)
- concat_transpose shader (delta-net dim-0)
- mmid_row_lists prepass + padded_N + use_row_lists (coexist with upstream's
  n_experts + hoist_row_ids; both push_constant fields kept in struct and
  mul_mm.comp / mul_mmq.comp dispatch tries row_lists first, then hoisted,
  then dense)
- MoE expert residency tracking + observability (CLI args, fields, debug)
- DFlash framework: decoder_laguna gate, embeddings_nextn post-final-layer
  lookup, dflash_conv/DFlash2 (selector + p_min)
- DSpark (Nemotron) attn_sinks
- Upstream's spec : fuse the DFlash encoder into the KV cache injection
  (ggml-org#27310) replaces the HEAD's two-step llama_encode + llama_decode path
  with a fused single llama_decode. Same encoder (fc + norm) runs inline
  as part of the decode graph. EAGLE3/MTP encode path (separate code in
  common_speculative_impl_draft_eagle3) is untouched.
- Laguna-S-2.1 model
- User isolation (user_id, conv_hash, slot affinity)
- SSD-backed KV cache and page manager
- Context checkpoint ring buffer with insertion-order eviction
- --lazy-mode (upstream rename of CachyLLama's --tensor-read-lazy)
- kv_unified_per_slot (upstream, additive)
- nvfp4 scales to attention ops (ggml-org#28000)
- quantize: row-slab stream to avoid thread starvation (ggml-org#27830)
- test-save-load-state across all architectures (ggml-org#27755)
- test-archs verbosity fix (ggml-org#28147)

Merge conflicts resolved (see ac01c2dc6 for full details - same resolutions
re-applied for the redo).

Verified:
- llama-server builds clean
- test-save-load-state passes (38s) after rebuild
- DSV4_HC_COMB shaders all pass
- Pre-existing failures unchanged (minimax-m3/01 numerical tolerance,
  test-recurrent-state-rollback)
fewtarius added a commit to fewtarius/CachyLLama that referenced this pull request Sep 1, 2026
@fewtarius
merge upstream/master (a14dba6..d086dbb): 76 commits
b43abe0 
Merge upstream commits preserving all CachyLLama features:
- CachyLLama Lightning Indexer shader + DSV4_HC pre/comb/post (kept over
  upstream cb30059's li_dict variant - c22b79e already removed the dup)
- FA dequant-once scratch with host-RAM safety gate (q8_0/q4_0/q4_1/q5_0/q5_1)
- concat_transpose shader (delta-net dim-0)
- mmid_row_lists prepass + padded_N + use_row_lists (coexist with upstream's
  n_experts + hoist_row_ids; both push_constant fields kept in struct and
  mul_mm.comp / mul_mmq.comp dispatch tries row_lists first, then hoisted,
  then dense)
- MoE expert residency tracking + observability (CLI args, fields, debug)
- DFlash framework: decoder_laguna gate, embeddings_nextn post-final-layer
  lookup, dflash_conv/DFlash2 (selector + p_min)
- DSpark (Nemotron) attn_sinks
- Upstream's spec : fuse the DFlash encoder into the KV cache injection
  (ggml-org#27310) replaces the HEAD's two-step llama_encode + llama_decode path
  with a fused single llama_decode. Same encoder (fc + norm) runs inline
  as part of the decode graph. EAGLE3/MTP encode path (separate code in
  common_speculative_impl_draft_eagle3) is untouched.
- Laguna-S-2.1 model
- User isolation (user_id, conv_hash, slot affinity)
- SSD-backed KV cache and page manager
- Context checkpoint ring buffer with insertion-order eviction
- --lazy-mode (upstream rename of CachyLLama's --tensor-read-lazy)
- kv_unified_per_slot (upstream, additive)
- nvfp4 scales to attention ops (ggml-org#28000)
- quantize: row-slab stream to avoid thread starvation (ggml-org#27830)
- test-save-load-state across all architectures (ggml-org#27755)
- test-archs verbosity fix (ggml-org#28147)

Verified:
- llama-server builds clean
- test-save-load-state passes (38s) after rebuild
- DSV4_HC_COMB shaders all pass
- Pre-existing failures unchanged (minimax-m3/01 numerical tolerance,
  test-recurrent-state-rollback)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC approved these changes

Assignees

No one assigned

Labels

merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. model Model specific

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JamePeng @CISC @ggerganov