Repository navigation
feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton - #174
Draft
HowardYangAI wants to merge 15 commits into
Draft
HowardYangAI wants to merge 15 commits into
HowardYangAI wants to merge 15 commits into
Conversation
Replace per-group Python loops with padded reshape reductions: INT4 key packing 0.091s -> 0.003s and NVFP4 key packing 0.680s -> 0.002s for 32k x 8 x 128 on GH200, with byte-identical payloads and 313 tests passing. Adds a codec registry and pack_paged/unpack_paged. (cherry picked from commit 606d15d)
- Dispatch pack_kv/unpack_kv through KV_CODECS - Expose group_axis/scale_shape on codecs and packed tensors - Share one grouped path for K and V - Mark decode and position checks as reference/debug paths
Pages that leave the latest kv_quantization_window tokens are packed into per-block slots; recent KV stays in a bounded BF16 staging pool. Attention decodes the batch's pages to BF16 and runs the existing backend. The memory plan charges packed pages, staging, the gather and padded codec scratch. Off by default. Async scheduling, CUDA graphs, speculative decoding, preemption, PD, prefix caching and the shared layout are rejected.
On CUDA, _gather_pages decodes INT4/NVFP4 pages straight into the BF16 attention input instead of calling the Torch reference codec per page. The kernel matches the reference bit for bit, including signed zeros, odd head_dim and head_dim smaller than the group size. CPU keeps the Torch path. Measured on L20 (8 seqs x 1024 prompt, 128 decode tokens, single run): INT4 49.3 -> 82.2 tok/s, NVFP4 2.0 -> 75.1 tok/s, BF16 baseline 105.2 tok/s.
Pack every page of a flush together, keeping one FP32 global scale per page. The result matches packing each page with nvfp4.pack bit for bit.
A fused Triton decode kernel decodes INT4/NVFP4 tiles in registers right before the dot product, instead of rewriting every history page to BF16 each step. Staged and packed pages are resolved per token through a per-page sources tensor. Prefill keeps the decode-to-BF16 path. Output matches decode-then-attend within one BF16 ulp. This step decodes NVFP4 with the same arithmetic as triton_decode. One attention layer, 16 heads, head_dim 128, 1152 tokens, RTX PRO 5000: INT4 fused 112 us vs decode+attend 164 us at 8 rows; 386 vs 770 us at 32 rows. NVFP4 fused 208 us vs decode+attend 161 us at 8 rows; 781 vs 788 us at 32 rows. INT4 is close to pure BF16 attention (108 / 377 us). NVFP4 is not yet a win; the next two commits fix that.
The reference decode computes (1 + 0.5 m) * exp2(e - 1), selects the subnormal case and multiplies in the sign: an exp2, several FP32 multiplies and selects per element of every tile. E2M1 has only eight magnitudes, and each is an exact FP32 whose bits follow directly from the 3-bit code: (m + 252) << 22 for m >= 2, 252 << 22 for m = 1, zero for m = 0, with code bit 3 moved to the FP32 sign bit. The values are identical bit for bit (including -0 for 0b1000), so the output does not change; tests pass unchanged. NVFP4 fused attention, one layer, 1152 tokens: 208 -> 155 us at 8 rows, 781 -> 570 us at 32 rows (two runs, within 0.2 us). INT4 is unaffected.
The global scale is one FP32 per page and layer, but the tile loaded it with a [tokens, dims] pointer, i.e. 128 identical loads per token, and divided per element. Loading it with a per-token pointer (masked by whether the token has any valid channel) and broadcasting removes those redundant loads. The division still happens per element with div_rn, so values are unchanged. NVFP4 fused attention, one layer, 1152 tokens: 155 -> 143 us at 8 rows, 570 -> 528 us at 32 rows. End to end on Ouro-1.4B, 8 seqs x 1024 prompt, 128 decode tokens, the full fused path reaches INT4 148.3 and NVFP4 151.9 tok/s (decode-then-attend: 137.7 and 134.6; BF16 160.6).
The fused kernel matched decode-then-attend exactly for INT4 but one output in 20480 differed for NVFP4 at head_dim 128. Traced it with three probes: - Decoding a tile in isolation and comparing against triton_decode gave 0 of 65536 differences for both K and V, so the arithmetic is not the cause. - Dumping the tile the kernel attends over showed keys and values identical to the decoded pages, but 16 of 32 scores off by one ulp. - TTGIR for both kernels showed the same reduction structure and the same number of splits, but different tile layouts: the reference and the INT4 path use sizePerThread [1, 8], NVFP4 uses [1, 16]. The cause is the scale width. A BF16 key element is two bytes and an INT4 scale is two bytes, so eight elements fill a 16-byte vector. An NVFP4 E4M3 scale is one byte, so Triton fills sixteen and assigns the tile [1, 16]; the FP32 score dot product then associates as a different tree. Tried and rejected: tl.max_contiguous on the scale index (compiles, does not change the choice), an explicit two-level sum reshaped into the reference's 8-then-16 tree (compiles to a different binary, still differs), and mul.rn through inline asm on the NVFP4 product (no effect). Document the caveat in the module docstring instead of forcing the layout.
The test relaxed every case to one BF16 ulp. Only NVFP4 needs that: INT4 reaches decode-then-attend exactly at all three shapes (dim 8, 65, 128), so assert equality there and keep the tolerance for NVFP4 alone, with the reason next to it.
triton_decode (the prefill path) still built E2M1 values with exp2 and a sign multiply, while the fused decode kernel had switched to assembling the FP32 bits directly. Both files also carried their own copy of the pool argument dispatch. Move _e2m1 and _pool_args into triton_decode and import them from the fused kernel. The decode kernel now takes the global scale as its own argument instead of overloading Second, and drops the unused TOKENS constexpr. The bit-built values are identical to the exp2 form, including -0 for code 0b1000; the bit-for-bit decode tests still pass (272 KV tests, --run-gpu). decode_pages_into, 512 pages x 64 tokens x 8 heads x 128, K and V, RTX PRO 5000 Blackwell, median of 5 do_bench runs, two alternating rounds: NVFP4 166.0 -> 157.6 us INT4 164.9 -> 164.9 us (unchanged code, control) The fused kernel is unchanged: 144 / 531 us at 8 / 32 rows before and after.
CI runs `ruff format --check .`, which failed on two lines this branch added: the hand-packed _packed_tile calls and the sources comprehension in _attention_plan. Apply the formatter. Shorten the module docstring's NVFP4 note to the cause and its effect; the probes and rejected workarounds are in a59304f's message. The test comment now points at the docstring instead of repeating it.
…kernel The fused kernel decodes each page once per row that reads it. In decode every row is a different sequence or loop plane, so that is once per page. But without packed prefill (the default), a prompt chunk runs as single-query rows that all read the same prefix, and every one of them re-decoded it: with 8 x 1024-token prompts each fused call had 1024 rows and _gather_pages was never called. _attention_plan already walks every row's pages, so it now counts page reads against unique pages and keeps the fused kernel only while each page is read fewer than twice on average. Other batches decode their pages once into BF16 and run the backend as before. Ouro-1.4B, 8 seqs x 1024 prompt, 128 decode tokens, RTX PRO 5000 Blackwell, warm Triton cache, two runs each (307f6b5 is the PyTorch-decode baseline): prefill s (307f6b5 / before / after) decode tok/s (before / after) INT4 3.00 / 4.89 / 2.73 156.3 / 158.0 NVFP4 18.2 / 6.62 / 2.90 153.1 / 154.4 Decode is unchanged, as expected: those batches still take the fused path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Wire KV quantization into the engine so Ouro fits more KV on the same GPU, for more requests at once or longer prompts. Ouro keeps KV for every loop, so 4 loops need 4x the KV. On a 5090 the BF16 KV pool is about 23 GiB while the weights are under 3 GB.
2. What is the highest-leverage variable for that outcome?
KV bytes per token: capacity is just how many tokens fit in the pool. The formats already exist from #166, so the leverage now is storing and reading packed pages in the engine without hurting accuracy.
3. How will this work move that variable rather than create busywork?
#166 only added the codecs, and nothing in the engine stores KV with them yet. This PR does, and the planner then fits 2.67x more KV tokens on a 5090 with INT4 or NVFP4. Decode now reads the packed pages in place with a Triton kernel, so on an RTX PRO 5000 INT4 and NVFP4 decode at 158 and 154 tok/s against about 167 for BF16 (8 requests, 1K prompts). With the PyTorch decode in the first version it was 59 and 3. On 200 held-out GSM8K questions INT4 was -2.5 pp (se 1.8) and NVFP4 +1.5 pp (se 2.1) against BF16. Neither is clearly real at this size, but I wouldn't call it lossless either. It stays off by default until it is checked on more models.
4. Who is the customer or user, and what do they actually need?
People serving Ouro-style looped models on one GPU who run out of KV room, the setting FlashLoop targets. They need to pack KV to fit more requests or longer prompts without losing much accuracy or speed, and nothing changes if they leave it off.
PR necessity and bugfix evidence
Yes. There is no way to store packed KV today. On #166 I said I'd keep that PR codec-only and do the integration separately; this is it. Feature, not a bugfix.
Purpose
Stacked on #166. Until that merges, this diff includes its commits; everything after e552a8a is new (307f6b5 plus ten kernel commits).
This wires the INT4/NVFP4 codecs into
KVCacheManager. It is off by default (kv_quantization=None). Default groups are G64 for INT4 and G16 for NVFP4.kv_quantization_windowtokens (default 64), it lives in a small BF16 staging pool.StageEngine._pack_kv). Each page is packed once per layer, so NVFP4 keeps one global scale per page and layer. On CUDA, NVFP4 pages of one call are encoded in one batched pass (nvfp4_batch.pack_batch), bit for bit equal to packing each page alone.packed_paged_attention, which reads staged BF16 pages and packed pages in place and decodes each tile in registers right before its dot product. No BF16 copy of the history is written.decode_pages_into) and call the backend. The switch is a page-read count_attention_planalready has: fused while each page is read fewer than twice on average.cu_seqlens_q) always takes the decode-once path.Memory plan:
staging_pages_bound: per request and plane, the pages left unpacked after packing (ceil((window + block_size - 1) / block_size)) plus one for new writes, and the step's tokens once, shared by all requests. The plan chargesmin(staging, blocks)BF16 pages.Not supported yet, and rejected with an error: async scheduling, CUDA graphs, speculative decoding, preemption and snapshots, PD transfer, prefix caching, the shared KV layout, models where not every cache layer is recurrent, and device-prepared metadata.
New options:
CacheConfig.kv_quantization,kv_quantization_group_size,kv_quantization_window,kv_quantization_staging_blocks, and the matching--kv-quantization*flags.Not in this PR: using the cross-loop residual from #166 in the cache, and the benchmark scripts I used for the numbers below. I can share the scripts and raw outputs.
Test Plan
vllm-rlt Version / Commit: 8e35940, on top of e552a8a (#166). The 5090 results further down are from 307f6b5, before the kernels.
pre-commit run --all-files--run-gpu: Triton decode vs the PyTorch reference, batched NVFP4 encode vs per-page encode, fused attention vs decode-then-attendpytest --run-gpu tests, compared with 307f6b5Test Result
RTX PRO 5000 Blackwell, 8e35940
RTX PRO 5000 72GB Blackwell (SM120), driver 580.126.09, torch 2.13.0+cu130, triton 3.7.1, triton backend, ByteDance Ouro-1.4B, block size 64.
Throughput, 8 requests x 1024-token prompts, 128 output tokens, 4 loops, warm Triton cache, mean of two runs:
BF16 does not touch this code; its four runs spread from 163 to 174 tok/s, so treat the BF16 columns as one number. Quantized decode is now within a few percent of BF16 but not faster: the kernel still does the BF16 attention work plus the decode.
Single layer, 16 heads, head_dim 128, 1152-token context: the fused kernel takes 112 us (INT4) and 144 us (NVFP4) at 8 rows against 160 us for decode-then-attend and 105 us for BF16 attention alone; at 32 rows 384 and 531 us against 763 to 771 and 368 to 374.
GSM8K (greedy, 4 loops, strict-match), questions correct:
RTX 5090, 307f6b5 (before the kernels)
RTX 5090 (SM120), torch 2.12.1+cu130, triton 3.7.1, triton backend, ByteDance Ouro-1.4B, block size 64.
Memory (
gpu_memory_utilization=0.9, 8 requests, each config filled to about 95% of its own capacity). The kernel commits do not change the memory plan, and I did not rerun this:At the same plan target the cache holds 2.67x more tokens. Every request finished and peak reserved stayed within the 28.22 GiB budget. Prompts were 3,709 tokens for BF16 and about 9,950 for the others, so the peaks do not show savings at equal load.
GSM8K (greedy, 4 loops, strict-match), questions correct:
Not run: FlashAttention backends (paged FA4 does not support SM120), GPUs other than these two, models other than Ouro-1.4B.