Skip to content

feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton - #174

Draft
HowardYangAI wants to merge 15 commits into
InferMatrixLab:mainfrom
HowardYangAI:feat/kv-quant-integration
Draft

HowardYangAI wants to merge 15 commits into
InferMatrixLab:mainfrom
HowardYangAI:feat/kv-quant-integration

Conversation

@HowardYangAI

@HowardYangAI HowardYangAI commented Nov 11, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Wire KV quantization into the engine so Ouro fits more KV on the same GPU, for more requests at once or longer prompts. Ouro keeps KV for every loop, so 4 loops need 4x the KV. On a 5090 the BF16 KV pool is about 23 GiB while the weights are under 3 GB.

2. What is the highest-leverage variable for that outcome?

KV bytes per token: capacity is just how many tokens fit in the pool. The formats already exist from #166, so the leverage now is storing and reading packed pages in the engine without hurting accuracy.

3. How will this work move that variable rather than create busywork?

#166 only added the codecs, and nothing in the engine stores KV with them yet. This PR does, and the planner then fits 2.67x more KV tokens on a 5090 with INT4 or NVFP4. Decode now reads the packed pages in place with a Triton kernel, so on an RTX PRO 5000 INT4 and NVFP4 decode at 158 and 154 tok/s against about 167 for BF16 (8 requests, 1K prompts). With the PyTorch decode in the first version it was 59 and 3. On 200 held-out GSM8K questions INT4 was -2.5 pp (se 1.8) and NVFP4 +1.5 pp (se 2.1) against BF16. Neither is clearly real at this size, but I wouldn't call it lossless either. It stays off by default until it is checked on more models.

4. Who is the customer or user, and what do they actually need?

People serving Ouro-style looped models on one GPU who run out of KV room, the setting FlashLoop targets. They need to pack KV to fit more requests or longer prompts without losing much accuracy or speed, and nothing changes if they leave it off.

PR necessity and bugfix evidence

Yes. There is no way to store packed KV today. On #166 I said I'd keep that PR codec-only and do the integration separately; this is it. Feature, not a bugfix.

Purpose

Stacked on #166. Until that merges, this diff includes its commits; everything after e552a8a is new (307f6b5 plus ten kernel commits).

This wires the INT4/NVFP4 codecs into KVCacheManager. It is off by default (kv_quantization=None). Default groups are G64 for INT4 and G16 for NVFP4.

  • A block id is a logical page and owns a packed slot in every layer. While a page is within the latest kv_quantization_window tokens (default 64), it lives in a small BF16 staging pool.
  • A page packs once every layer of its loop plane has written past its end plus the window. Packing runs between synchronous steps (StageEngine._pack_kv). Each page is packed once per layer, so NVFP4 keeps one global scale per page and layer. On CUDA, NVFP4 pages of one call are encoded in one batched pass (nvfp4_batch.pack_batch), bit for bit equal to packing each page alone.
  • Attention, on CUDA with the Triton backend:
    • Decode batches go through packed_paged_attention, which reads staged BF16 pages and packed pages in place and decodes each tile in registers right before its dot product. No BF16 copy of the history is written.
    • That kernel decodes a page once per row that reads it, so batches whose rows share pages (prompt chunks, which all read the same prefix) instead decode each page once into BF16 with a Triton kernel (decode_pages_into) and call the backend. The switch is a page-read count _attention_plan already has: fused while each page is read fewer than twice on average.
    • Packed prefill (cu_seqlens_q) always takes the decode-once path.
  • Other devices and backends keep the PyTorch reference decode.
  • Exactness: the Triton decode equals the PyTorch reference bit for bit for both formats. The fused kernel equals decode-then-attend bit for bit for INT4. For NVFP4 it decodes the same values, but the one-byte E4M3 scale makes Triton lay the tile out differently, so the FP32 score sum can differ by one ulp in rare outputs at head_dim 128. The tests allow exactly that.
  • Pages also pack during prefill, so later prompt chunks read packed pages. FlashLoop waits until the prompt ends. I kept packing during prefill so staging depends on the step size, not the prompt length. In the 5090 runs below, waiting was not better on held-out questions.

Memory plan:

  • Each block id costs its packed bytes, plus one BF16 layer of K/V for the gather.
  • Staging is bounded by staging_pages_bound: per request and plane, the pages left unpacked after packing (ceil((window + block_size - 1) / block_size)) plus one for new writes, and the step's tokens once, shared by all requests. The plan charges min(staging, blocks) BF16 pages.
  • Codec scratch is sized for the padded shape, since V pads channels to a multiple of the group size.

Not supported yet, and rejected with an error: async scheduling, CUDA graphs, speculative decoding, preemption and snapshots, PD transfer, prefix caching, the shared KV layout, models where not every cache layer is recurrent, and device-prepared metadata.

New options: CacheConfig.kv_quantization, kv_quantization_group_size, kv_quantization_window, kv_quantization_staging_blocks, and the matching --kv-quantization* flags.

Not in this PR: using the cross-loop residual from #166 in the cache, and the benchmark scripts I used for the numbers below. I can share the scripts and raw outputs.

Test Plan

vllm-rlt Version / Commit: 8e35940, on top of e552a8a (#166). The 5090 results further down are from 307f6b5, before the kernels.

  • pre-commit run --all-files
  • New kernel tests with --run-gpu: Triton decode vs the PyTorch reference, batched NVFP4 encode vs per-page encode, fused attention vs decode-then-attend
  • Full pytest --run-gpu tests, compared with 307f6b5
  • Ouro-1.4B decode throughput against 307f6b5 and BF16
  • GSM8K against BF16 KV on a held-out sample and the fixed case

Test Result

RTX PRO 5000 Blackwell, 8e35940

RTX PRO 5000 72GB Blackwell (SM120), driver 580.126.09, torch 2.13.0+cu130, triton 3.7.1, triton backend, ByteDance Ouro-1.4B, block size 64.

  • pre-commit: Ruff lint and format pass. CPU tests: 1190 passed, 761 skipped.
  • KV tests on GPU (8 files, including the new kernel tests): 421 passed.
  • Full GPU suite: 71 failed, 163 errors, and every one of those 234 test IDs also fails on 307f6b5 on this machine. 307f6b5 had one more failure, an allocator-residue memory test, while both suites ran at the same time on neighbouring GPUs.

Throughput, 8 requests x 1024-token prompts, 128 output tokens, 4 loops, warm Triton cache, mean of two runs:

KV decode tok/s, 307f6b5 decode tok/s, 8e35940 time to first token, 307f6b5 time to first token, 8e35940
BF16 172 167 2.35 s 2.34 s
INT4 59.2 158.0 3.00 s 2.73 s
NVFP4 3.0 154.4 18.2 s 2.90 s

BF16 does not touch this code; its four runs spread from 163 to 174 tok/s, so treat the BF16 columns as one number. Quantized decode is now within a few percent of BF16 but not faster: the kernel still does the BF16 attention work plus the decode.

Single layer, 16 heads, head_dim 128, 1152-token context: the fused kernel takes 112 us (INT4) and 144 us (NVFP4) at 8 rows against 160 us for decode-then-attend and 105 us for BF16 attention alone; at 32 rows 384 and 531 us against 763 to 771 and 368 to 374.

GSM8K (greedy, 4 loops, strict-match), questions correct:

KV held-out (200) vs BF16, paired held-out without GSM8K-100 (187) fixed case (87)
BF16 135 126 59
INT4 130 -2.5 pp (se 1.8) 120, -3.2 pp (se 1.8) 60
NVFP4 138 +1.5 pp (se 2.1) 128, +1.1 pp (se 2.1) 63
  • Held-out: 200 test questions outside the fixed case, ranked by a seeded hash, with at most 512 new tokens. 13 of them are in the GSM8K-100 set the fixed case came from, so the 187 column drops them. This is a different and larger sample than the 5090 one below, and a different GPU, so the two tables are not directly comparable.
  • Neither difference is clearly real at this size. That is not the same as no loss.
  • Kernels vs decode-then-attend, whole runs: INT4 produced the same tokens on all 87 fixed questions. NVFP4 gave the same tokens on 76 of 87 (both 63 correct) and on 149 of 200 held-out questions (137 vs 138 correct), which is the one-ulp score difference compounding over long generations.
  • Per question on the fixed case (one question at a time): 4.8 s BF16, 5.6 s INT4, 6.1 s NVFP4.

RTX 5090, 307f6b5 (before the kernels)

RTX 5090 (SM120), torch 2.12.1+cu130, triton 3.7.1, triton backend, ByteDance Ouro-1.4B, block size 64.

  • pre-commit: Ruff lint and format pass. CPU tests: 1190 passed, 733 skipped.
  • New tests on GPU: 42 passed.
  • Full GPU suite: 71 failed, 163 errors. Same 234 test IDs and same errors as e552a8a: 163 are native kernels not built, 56 are paged FA4 not supported on SM120.
  • Fake quantization: token IDs match on every question checked (INT4 6, NVFP4 4), so paging, staging and the gather add no error of their own there.

Memory (gpu_memory_utilization=0.9, 8 requests, each config filled to about 95% of its own capacity). The kernel commits do not change the memory plan, and I did not rerun this:

KV blocks KV token slots peak reserved
BF16 1969 31,504 28.22 GiB
INT4 5255 84,080 25.03 GiB
NVFP4 5250 84,000 25.95 GiB

At the same plan target the cache holds 2.67x more tokens. Every request finished and peak reserved stayed within the 28.22 GiB budget. Prompts were 3,709 tokens for BF16 and about 9,950 for the others, so the peaks do not show savings at equal load.

GSM8K (greedy, 4 loops, strict-match), questions correct:

KV held-out (107) vs BF16, paired fixed case (87)
BF16 74 62
INT4 71 -2.8 pp (se 3.1) 58
NVFP4 72 -1.9 pp (se 3.0) 65
INT4, packing after the prompt 67 -6.5 pp (se 2.7) 63
  • Held-out: 120 test questions outside the fixed case, minus 13 that were in the GSM8K-100 set the fixed case came from. The fixed 87 were picked after looking at earlier results, so I only use them as a regression check.
  • Packing after the prompt (as FlashLoop does) was not better on the held-out set: -3.7 pp (se 2.6) against INT4 as in this PR. It was +5.7 pp on the fixed case. The two sets disagree, so this shows waiting doesn't clearly help, not that the two policies are the same.
  • NVFP4 hit the 1024-token limit on 2 held-out questions. Both are repetition loops on questions BF16 also gets wrong.
  • Speed with the PyTorch decode (fixed 87 questions): 6.7 s per question for BF16, 10.3 s for INT4, 32 s for NVFP4. The kernels above replace that path.

Not run: FlashAttention backends (paged FA4 does not support SM120), GPUs other than these two, models other than Ouro-1.4B.

howardforpr and others added 15 commits November 6, 2026 18:44
Replace per-group Python loops with padded reshape reductions: INT4 key packing 0.091s -> 0.003s and NVFP4 key packing 0.680s -> 0.002s for 32k x 8 x 128 on GH200, with byte-identical payloads and 313 tests passing. Adds a codec registry and pack_paged/unpack_paged.

(cherry picked from commit 606d15d)
- Dispatch pack_kv/unpack_kv through KV_CODECS
- Expose group_axis/scale_shape on codecs and packed tensors
- Share one grouped path for K and V
- Mark decode and position checks as reference/debug paths
Pages that leave the latest kv_quantization_window tokens are packed into
per-block slots; recent KV stays in a bounded BF16 staging pool. Attention
decodes the batch's pages to BF16 and runs the existing backend. The memory
plan charges packed pages, staging, the gather and padded codec scratch.

Off by default. Async scheduling, CUDA graphs, speculative decoding,
preemption, PD, prefix caching and the shared layout are rejected.
On CUDA, _gather_pages decodes INT4/NVFP4 pages straight into the BF16 attention input instead of calling the Torch reference codec per page. The kernel matches the reference bit for bit, including signed zeros, odd head_dim and head_dim smaller than the group size. CPU keeps the Torch path.

Measured on L20 (8 seqs x 1024 prompt, 128 decode tokens, single run): INT4 49.3 -> 82.2 tok/s, NVFP4 2.0 -> 75.1 tok/s, BF16 baseline 105.2 tok/s.
Pack every page of a flush together, keeping one FP32 global scale per page. The result matches packing each page with nvfp4.pack bit for bit.
A fused Triton decode kernel decodes INT4/NVFP4 tiles in registers right before the dot product, instead of rewriting every history page to BF16 each step. Staged and packed pages are resolved per token through a per-page sources tensor. Prefill keeps the decode-to-BF16 path. Output matches decode-then-attend within one BF16 ulp.

This step decodes NVFP4 with the same arithmetic as triton_decode. One attention layer, 16 heads, head_dim 128, 1152 tokens, RTX PRO 5000:
  INT4  fused 112 us vs decode+attend 164 us at 8 rows; 386 vs 770 us at 32 rows.
  NVFP4 fused 208 us vs decode+attend 161 us at 8 rows; 781 vs 788 us at 32 rows.
INT4 is close to pure BF16 attention (108 / 377 us). NVFP4 is not yet a win; the next two commits fix that.
The reference decode computes (1 + 0.5 m) * exp2(e - 1), selects the subnormal case and multiplies in the sign: an exp2, several FP32 multiplies and selects per element of every tile. E2M1 has only eight magnitudes, and each is an exact FP32 whose bits follow directly from the 3-bit code: (m + 252) << 22 for m >= 2, 252 << 22 for m = 1, zero for m = 0, with code bit 3 moved to the FP32 sign bit. The values are identical bit for bit (including -0 for 0b1000), so the output does not change; tests pass unchanged.

NVFP4 fused attention, one layer, 1152 tokens: 208 -> 155 us at 8 rows, 781 -> 570 us at 32 rows (two runs, within 0.2 us). INT4 is unaffected.
The global scale is one FP32 per page and layer, but the tile loaded it with a [tokens, dims] pointer, i.e. 128 identical loads per token, and divided per element. Loading it with a per-token pointer (masked by whether the token has any valid channel) and broadcasting removes those redundant loads. The division still happens per element with div_rn, so values are unchanged.

NVFP4 fused attention, one layer, 1152 tokens: 155 -> 143 us at 8 rows, 570 -> 528 us at 32 rows. End to end on Ouro-1.4B, 8 seqs x 1024 prompt, 128 decode tokens, the full fused path reaches INT4 148.3 and NVFP4 151.9 tok/s (decode-then-attend: 137.7 and 134.6; BF16 160.6).
The fused kernel matched decode-then-attend exactly for INT4 but one output
in 20480 differed for NVFP4 at head_dim 128. Traced it with three probes:

- Decoding a tile in isolation and comparing against triton_decode gave 0
  of 65536 differences for both K and V, so the arithmetic is not the cause.
- Dumping the tile the kernel attends over showed keys and values identical
  to the decoded pages, but 16 of 32 scores off by one ulp.
- TTGIR for both kernels showed the same reduction structure and the same
  number of splits, but different tile layouts: the reference and the INT4
  path use sizePerThread [1, 8], NVFP4 uses [1, 16].

The cause is the scale width. A BF16 key element is two bytes and an INT4
scale is two bytes, so eight elements fill a 16-byte vector. An NVFP4 E4M3
scale is one byte, so Triton fills sixteen and assigns the tile [1, 16]; the
FP32 score dot product then associates as a different tree.

Tried and rejected: tl.max_contiguous on the scale index (compiles, does not
change the choice), an explicit two-level sum reshaped into the reference's
8-then-16 tree (compiles to a different binary, still differs), and mul.rn
through inline asm on the NVFP4 product (no effect). Document the caveat in
the module docstring instead of forcing the layout.
The test relaxed every case to one BF16 ulp. Only NVFP4 needs that: INT4
reaches decode-then-attend exactly at all three shapes (dim 8, 65, 128), so
assert equality there and keep the tolerance for NVFP4 alone, with the
reason next to it.
triton_decode (the prefill path) still built E2M1 values with exp2 and a
sign multiply, while the fused decode kernel had switched to assembling the
FP32 bits directly. Both files also carried their own copy of the pool
argument dispatch.

Move _e2m1 and _pool_args into triton_decode and import them from the fused
kernel. The decode kernel now takes the global scale as its own argument
instead of overloading Second, and drops the unused TOKENS constexpr.

The bit-built values are identical to the exp2 form, including -0 for code
0b1000; the bit-for-bit decode tests still pass (272 KV tests, --run-gpu).

decode_pages_into, 512 pages x 64 tokens x 8 heads x 128, K and V, RTX PRO
5000 Blackwell, median of 5 do_bench runs, two alternating rounds:

  NVFP4  166.0 -> 157.6 us
  INT4   164.9 -> 164.9 us  (unchanged code, control)

The fused kernel is unchanged: 144 / 531 us at 8 / 32 rows before and after.
CI runs `ruff format --check .`, which failed on two lines this branch
added: the hand-packed _packed_tile calls and the sources comprehension in
_attention_plan. Apply the formatter.

Shorten the module docstring's NVFP4 note to the cause and its effect; the
probes and rejected workarounds are in a59304f's message. The test comment
now points at the docstring instead of repeating it.
…kernel

The fused kernel decodes each page once per row that reads it. In decode
every row is a different sequence or loop plane, so that is once per page.
But without packed prefill (the default), a prompt chunk runs as single-query
rows that all read the same prefix, and every one of them re-decoded it:
with 8 x 1024-token prompts each fused call had 1024 rows and _gather_pages
was never called.

_attention_plan already walks every row's pages, so it now counts page reads
against unique pages and keeps the fused kernel only while each page is read
fewer than twice on average. Other batches decode their pages once into BF16
and run the backend as before.

Ouro-1.4B, 8 seqs x 1024 prompt, 128 decode tokens, RTX PRO 5000 Blackwell,
warm Triton cache, two runs each (307f6b5 is the PyTorch-decode baseline):

          prefill s (307f6b5 / before / after)   decode tok/s (before / after)
  INT4    3.00 / 4.89 / 2.73                     156.3 / 158.0
  NVFP4   18.2 / 6.62 / 2.90                     153.1 / 154.4

Decode is unchanged, as expected: those batches still take the fused path.
@HowardYangAI HowardYangAI changed the title feat(kv): store INT4/NVFP4 KV pages in the cache manager feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants