[Benchmark] IQ3_S on RTX 5070 + RTX 5060 Ti: decode 73.3 → 112.5 tok/s, prefill to 2,896, and the prompt-chunk clamp that costs 69 % of prefill past 704k #4603

Benchmark report, no engine change. One PC, two mismatched Blackwell cards, IQ3_S of Qwen3.8-Flash-Next at 262,144 tokens, expert arena in RAM. Measured on 2026-10-29 with a self-built engine 0.1.52 (CMAKE_CUDA_ARCHITECTURES=120).

Hardware: RTX 5070 12 GB (48 SMs, 2.54 GHz) + RTX 5060 Ti 16 GB (36 SMs, 2.60 GHz), both sm_120, both display_active: Disabled (so --vram-reserve-mib 0 is correct here), engine PCIe probe 28.7 / 28.9 GB/s. Ryzen 9 9950X3D2, 16 cores / 32 threads, AVX-512, one socket. 125.5 GiB RAM, commit limit 219.3 GiB. Windows 11, CUDA 13.

Model: Qwen3.8-Flash-Next IQ3_S, native mode (--native + --ple-gguf over two GGUF shards), pack packs/iq3_s = dense.bin 1.43 GiB + experts.bin 46.84 GiB renamed to .off so the run stays in the arena (see §5). Expert profile 24,576 ranked pairs. Context 262,144, --kv int8, --spec 4 --spec-min-p 0.5, --mtp, --vision, --expert-cache auto, --prefill auto, --host-core sibling, --vram-reserve-mib 0. Cards "gpu": [1, 0] — 5060 Ti holds layers 0–26, 5070 holds 27–47 plus the head and the draft verification.

At a glance: one card against two

Same profile everywhere (layer_split 27, STRATA_STAGE_TRIM=1, STRATA_ARENA_PIN_GIB=52, --vram-reserve-mib 0), same Laravel/Vue code prompt, 700 generated tokens, warm-up discarded. The only difference between rows is which cards sit in "gpu" and how much context is allocated.

GPU load:

Rows are grouped by context, and within a context the order is always 5060 Ti → 5070 → both cards, so the effect of adding the second card reads downwards. chunk is in the table because it decides prefill; where it is 8192 the prompt is read in one pass. Both bars are only 6 blocks wide, and each is scaled to the range it actually spans: decode starts at 65 tok/s and adds a block per 8 (so 71 tok/s is 1 block, 112 tok/s is 6); prefill adds a block per 500 (6 blocks = 3,000). The ratio between rows is visible without reading numbers, and the prefill column makes the chunk clamp obvious - the 256k RTX 5070 row fills 3 blocks against 6 for the two-card rows.

The same numbers as charts

Decode against context — where the curves cross is where the second card starts to pay:

xychart-beta
    title "Decode, tok/s (code prompt, 700 tokens)"
    x-axis ["32k", "64k", "128k", "192k", "256k"]
    y-axis "tok/s" 60 --> 120
    line "RTX 5060 Ti" [90.8, 87.9, 84.9, 81.4, 79.4]
    line "RTX 5070" [87.5, 89.5, 84.0, 77.8, 71.5]
    line "both cards" [115.3, 114.7, 112.5, 111.6, 109.8]

Prefill against context — flat until 256k, then the chunk clamp starts to bite:

xychart-beta
    title "Prefill, tok/s on a 54k prompt"
    x-axis ["32k", "64k", "128k", "192k", "256k"]
    y-axis "tok/s" 900 --> 3000
    line "RTX 5060 Ti" [1811, 1802, 1774, 1795, 1796]
    line "RTX 5070" [2433, 2436, 2238, 1877, 1101]
    line "both cards" [2646, 2882, 2876, 2878, 2896]

What the two charts say at a glance:

• Decode: the two single cards are within a few tok/s of each other up to 128k — the 5060 Ti is ahead at 32k, the 5070 at 64k. Only at 256k does the 12 GB card fall away (71.5 against 79.4), because its cache drops to 703 slots.
• Both cards are not just "one card plus one card" — their peak is at the smallest context (115.3 at 32k, 4,997 slots), and every step up costs a little: 114.7 at 64k, 112.5 at 128k, down to 109.8 at 256k. At 32k the two-card row is 27 % above the faster single card. This is the opposite of the single-card lines, which gain from a smaller window for a different reason (a smaller KV cache leaves more expert-cache slots), so both effects point the same way: on this rig, decode is bought with expert-cache slots.
• The 32k and 64k two-card rows are not measured the same way as the rest — the desktop app adds its own --batch there, which is why §10 exists. Those rows are the one-request figures.
• Prefill is where the cards differ most: the 5070 leads the 5060 Ti by 35 % from 32k to 128k, then collapses to 1,101 at 256k. The two-card line is flat at ~2,890 all the way to 256k and only bends past 512k (§1.5c).

Every numeric column shows one number per row: where two sessions of the same configuration measured 88.6 and 93.1 decode, the row shows their midpoint, 90.8, and hit rate likewise becomes an integer midpoint (76–84 % → 80 %). The unrounded pairs are kept in §1.3, where the spread itself is the point — it is the column that explains decode.

Only the usable configuration for each context appears. Where a setting measured worse than the alternative at the same window — the default K=27 at 768k (prefill 912 tok/s), K=30 or K=34 at the same points — it is left out of this table and kept in §1.5b/§1.5c as part of the reasoning, so this table never lists a row you would not actually choose.

The sweet spot per card: one configuration each

Each single card has exactly one context where decode, chunk, prefill and hit rate are all at their best together. Picking it means no compromise on either end — the chunk is a full 8,192 and the cache is still large enough to hold the working set.

For comparison, the same cards run into a wall when the window grows past those points:

So the two cards answer q4_0 differently: on the 5070 it is clearly worth it at 192k (it un-clamps the chunk and adds 17 % prefill), on the 5060 Ti it trades prefill for decode and is a wash. Which is consistent with everything above — q4_0 pays when the chunk is clamped, and the 16 GB card is not clamped at 192k.

At a glance, part 2: where the context stops and where it is fastest

The model is trained at 262,144 but the stack does not stop there — --max-context takes more and RoPE keeps working. Walked up on two cards until it refused:

• The ceiling is 1,048,576 — 4× the trained window — but only with a re-balanced split. At the default K=27 the run dies at 768k with mtp: the K/V state does not fit (561 MiB of 12,199 MiB VRAM free on this GPU). The 5070 holds layers 27–47 plus the head plus the MTP drafter, and the drafter's K/V state is what runs out — not the model, not the KV cache, not correctness. Moving more layers to the 16 GB card fixes it: K=32 loads 1M, K=27 cannot.
• But the fix has its own optimum. At 1M, K=32 gives 1,091 slots and 75.2 tok/s; K=36 leaves only 502 slots and 68.3 tok/s (−9 %). Pushing layers further onto the big card starves the expert cache on the small one. K=32 is the balance.
• Prefill is what really breaks past 262k, and it breaks monotonically. The chunk shrinks step by step as the window grows, and prefill follows it down — every point measured:

Up to 262k the chunk stays at 8,192 and prefill is flat within 1 % — that is where the usable range ends for prompt-heavy work. From 512k the clamp starts and each step costs 8–17 %; by 768k it is two thirds gone. The 1M window is decode-only in practice (75.2 tok/s at K=32, but 1,613 tok/s prefill against 2,882 at 64k), and at K=36 it is unusable for prompts entirely (441 tok/s, a 512-token chunk in an 8-slot ring).

* The 32k row is marked because a 54k prompt does not fit a 32,768 window: that one row is a 30,579-token prompt (800 paragraphs instead of 1,400). The difference is real and not a measurement artefact — a shorter prompt has fewer chances to hit the expert cache and more per-token overhead at the start — so the 32k prefill figure is not directly comparable with the rows below it. Everything from 64k up is the same 54k prompt.

1.5b. The chunk is a function of K, not of the total slot count

The question that exposed this: why does 1M with K=32 get a bigger chunk (2,816) than the smaller 704k window at K=27 (2,560), despite having fewer total slots (1,091 against 3,089)? Because the chunk is set by CUDA1's share of the cache, and K decides how many layers — and therefore how much cache — CUDA1 holds. Total slots are irrelevant to it.

Measured across every point where both numbers are in the log:

The rule: CUDA1 must lend ~1,692 slots for an 8,192-token chunk, and it may lend 90 % of what it holds minus 128. So the chunk grows when K pushes more layers (and thus more cache) onto the 16 GB card — regardless of what happens to the total.

That is why the naive reading "more slots = bigger chunk" fails here, and why the same 768k window behaves completely differently at K=27 and K=32.

1.5c. The same 768k window, four configurations — a 2.9× swing in prefill

This is the clearest demonstration of the whole report: one window, one model, four profiles, prefill from 912 to 2,621 tok/s.

Three things follow:

• The default K=27 is the worst choice at 768k — not because the window is too big, but because it leaves CUDA1 the least cache to lend. The 912 tok/s figure I reported earlier was a property of the split, not of the window.
• q4_0 wins again and by the widest margin yet: +187 % prefill and +19 % decode over the same window at K=27, with the most slots of any configuration (3,758). It needs no layer moving at all.
• K=32 is the best int8 option (+144 % prefill, +8 % decode) but costs 857 slots (2,908 → 2,051), so it is a real trade; q4_0 is not.

Practical rule, now measured four times: when prefill matters at a large context, change the KV type before touching the split. q4_0 adds slots to both cards at once; moving layers only shifts them, and the card that gives them up pays in decode and hit rate.

1.6. Why 512k gets 6,656 and not 8,192 — and the two fixes that restore it

The engine says it exactly, in the WARNING quoted in §4:

WARNING: prompt chunk 6656 tokens, not 8192: CUDA1 (RTX 5070) has 1710 expert-cache slots,
and a 8192-token chunk borrows 1710 of them (it must keep 128, and lend at most 90%)

The arithmetic: 4,967 total slots in the pool, but only 1,710 belong to CUDA1, and an 8,192-token chunk wants 1,692 of them. CUDA1 must keep 128 and may lend at most 90 % — so 1,710 × 0.90 − 128 = 1,411 usable, short of 1,692. The scan steps down to 6,656. The clamp is set by the SMALLEST card's share of the cache, not by the total — the same #1616 mechanism as in §4.

Two levers restore the full chunk, and both were measured at 512k:

• Moving layers to the 16 GB card works mechanically (K=30 → 7,424, K=32 → full 8,192) but loses more than it gains: K=32 restores the chunk and still measures slower than the clamped baseline — 2,445 prefill against 2,665, and 90.9 decode against 95.6. Fewer slots on the 12 GB card cost more than the bigger chunk returns.
• q4_0 is the right fix: it frees VRAM without moving any layers, the cache grows to 4,201 slots, CUDA1 gets enough to lend, and everything improves — chunk back to 8,192, prefill 2,766.6 (+3.8 % over baseline), decode 103.6 (+8.4 %), hit rate 94–95 %.

So the answer to "why not 8,192" is: the second card's cache share caps the loan. Raising K moves the cap but starves the cache; halving the KV memory raises the cache and removes the cap for free. On this profile the rule is consistent with §1.4 and §1.5 — when the chunk is clamped, the fix is more slots, and q4_0 is the only lever that adds slots without taking anything away.

• The smaller the window, the faster two cards decode, which is the opposite of what the single-card lines do and for the same underlying reason — money is expert-cache slots. Pinned to one request at a time: 115.3 at 32k (4,997 slots), 114.7 at 64k (4,907), 112.5 at 128k (4,725), 111.6 at 192k (4,542), 109.8 at the trained 262k (4,361). Every 140 slots is worth about 0.3 tok/s.
• The real cost starts above 262k. From 32k to 262k the curve loses 4.8 %; from 262k to 704k it falls 19 %, and the expert cache loses 1,272 slots.
• 128k used to look like the sweet spot (+12 % over 32k, +2.5 % over 262k). It was an artefact of the app's slot policy — 32k and 64k carried --batch 3 and --batch 2 (§10). With one request at a time 128k is simply the middle of a monotone line, and 32k is the fastest point.

Hit rate is the column that explains the decode column. Among the single-card points, ordered by slots, it climbs: 703 → 51–56 %, 1,317 → 61–67 %, 2,054 → 69–74 %, 2,333 → 71–76 %, 3,105 → 76–82 %, 3,491 → 80–84 %, 3,685 → 76–84 %. Among the two-card points it is flat-high and ordered: 4,997 → 95–98 %, 4,907 → 93–97 %, 4,361 → 96–98 %. A miss sends that expert across PCIe (10–12 % of all routed experts at the worst point), so an "almost empty" expert cache is not a neutral state — it is the dominant single-card decode cost. See §1.3.

One anomaly worth naming: on the single cards, 3,685 slots give 76–84 %, fewer than the 3,491 slots' 80–84 % — the hit rate is not a pure function of the slot count. What varies is which experts are cached: the profile ranks pairs by routed mass, so on a different card (and with a layer split) the same slot count covers a different share of the traffic. Treat the column as a strong trend, not a lookup table.

Read the chunk column before comparing prefill rows. Where it is 8192 the prompt is read in one pass; the two 2,048 / 6,144 rows are the same card at the same context, and the whole 1,101 → 2,041 prefill difference comes from that clamp, not from the KV arithmetic — see §1.4.

• A full 262,144 window fits on EITHER card alone — this surprised me, and my first draft of this report wrongly claimed one card stops at 128k. Verified on both: 78.2 tok/s on the 5060 Ti, 76.8 on the 5070 with q4_0.
• Two cards win everywhere: +40 % decode over the best single card at 256k (109.8 vs 78.2), and prefill 2,896 against 1,796–2,041. At 32k with one request at a time the same gap is +30 % (115.3 vs 88.6) with two independent sources of gain on top of each other — a shared expert cache and more than double the slots.
• The cards swap roles. The 5070 is the prefill card — 2,433 vs 1,811 at 32k, i.e. +34 % from 48 SMs against 36 at nearly equal clocks; the prompt path is compute-bound. The 5060 Ti is the decode card — 88.6 vs 84.6 at 32k despite fewer SMs, because its 16 GB buys 79 % more expert-cache slots (3,685 vs 2,054); decode with MTP is bound by cache hits, not compute.
• On 16 GB, the full window costs 12 % decode and nothing in prefill (88.6 → 78.2, 1,811 → 1,796). On 12 GB it nearly squeezes the expert cache out: 703 slots and the prompt halves to 1,101.
• --kv q4_0 is the fix for the 12 GB card at 256k: 703 → 1,317 slots, decode 71.7 → 76.8, prefill 1,101 → 2,041 (+85 %). See §1.5 for why the same flag hurts on the two-card 262,144 profile — the sign depends on whether the KV cache is squeezing the expert cache out.

An aside on the two-card rows, because the first numbers looked backwards — 98.7 tok/s at 32k against 109.8 at 256k means the smaller context decoded slower, which no KV-cache argument explains. I first blamed the 0 MiB of VRAM free warning, which is printed at 32k and is a real hazard on its own. That was wrong: the cause was the app's own --batch 3, worth 14.6 tok/s there (§10). With the slot count pinned to one request, 32k is the fastest point on two cards (115.3) and the curve is monotone, exactly as the slot arithmetic predicts:

# 32k, one request at a time: more free VRAM and more slots than any larger window
expert cache auto: 12.39 GiB free ... -> 4,997 slots
prompt chunk auto: 8192 tokens, a 384-slot ring

# 256k: the KV cache has taken its share, so 10.81 GiB is left for the cache
expert cache auto: 10.81 GiB free ... -> 4,361 slots

# 32k, the app's default: --batch 3 takes 0.89 GiB back
--batch 3: the slot sessions take 0.89 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
expert cache auto: 11.50 GiB free ... -> 4,636 slots

With a small context nothing stops the expert cache from taking every last MiB, and the driver starts paging GPU memory — which costs more than the extra 275 slots return. This is the trap in reverse: on this rig --vram-reserve-mib 0 is right for the 262,144 profile but wrong for small contexts. Adding a 768 MiB reserve at 32k raised free VRAM to 496 MiB and moved decode to 93.7 — but it also took 303 slots (4,636 → 4,333) on both cards, so it did not recover the loss either. That point was taken before I pinned the slot count, so 4,636 is the --batch 3 figure: it is valid as an A/B inside that session (reserve added, nothing else), not as an absolute 32k number. If you run a small context on this machine, measure the reserve rather than assuming 0.

Result: decode 73.3 → 115.3 tok/s (best point, 32k on both cards) and 2,481 → 2,896 tok/s prefill at 262k (2,882 at 64k); a 1,048,576-token window that loads with a re-balanced split; and --kv q4_0 that is worth +187 % prefill at 768k but a −2.1 % regression at 262k. The main settings, each tested alone:

1. STRATA_ARENA_PIN_GIB: the 8 GiB cap is the engine's default, not a WDDM limit

On a layer split under WDDM the engine registers at most 8 GiB of the host arena:

cudaHostRegister limited to 8 GiB by the engine (multi-GPU under WDDM, or remote experts); 8 slices pinned (7 GiB)

An explicit value overrides it. On IQ3_S the arena is 46.84 GiB and 40 GiB of it registers cleanly:

STRATA_ARENA_PIN_GIB=40: the expert arena's pinning is capped
cudaHostRegister limited to 40 GiB by the engine ...; 41 slices pinned (39 GiB)

The interesting part: there is a threshold between 46 and 48. Below it the engine cuts the arena into ~1 GiB slices and registers them one at a time (the WDDM per-registration limit):

STRATA_ARENA_PIN_GIB=40: the expert arena's pinning is capped
cudaHostRegister limited to 40 GiB by the engine ...; 41 slices pinned (39 GiB)

Above it, the whole arena registers in one go:

strata generate: expert arena: cudaHostRegister PORTABLE ok; large pages (2097152 B)

No per-slice registration overhead, and the prefill gains a stable +7.0 % (2,881–2,917 against 2,682–2,704 at 40; three runs per point, spread inside a point 15–35 tok/s). Decode gains 1–4 %, which is at the edge of noise — on prose it is invisible (93.1 vs 92.9).

Two things worth knowing: the expert-cache slot count does not move at all (4,361 at every value) because the arena pin and the expert cache live in different pools — the gain is purely the registration path; and past 52 nothing improves (54/56/60/64 keep PORTABLE but the prefill starts giving back). 52 is the value.

This is the mirror image of the NVFP4 finding, where 54 was a −9.9 % regression: there, more slicing meant more VRAM in paging. Here 48+ switches the mechanism instead.

0 ("pin everything") buys nothing: the driver's ceiling on this box is RAM/2 − 8 GiB ≈ 54 GiB, so the engine stops at the same 39 GiB and keeps slicing. Pin by number, not by 0.

The flag applies only in arena mode: with --resident-experts there is no arena to pin.

1.3. "Just leave the expert cache nearly empty" — it costs 40 % of decode

Measured on every point of the at-a-glance table (5 runs each, the range is the run-to-run spread of the log line decode expert cache hit rate):

The old four-point version of this section said the relationship was smooth. Across the two-card points it is: 4,997 → 96.6 %, 4,907 → 96.3 %, 4,361 → 96.8 %, and each step of ~140 slots is worth about 0.3 tok/s of decode. Across the single-card points it is monotonic but not a function of the slot count alone: 3,685 scores lower than 3,491 (76–84 % against 80–84 %). What differs is which experts are held — the profile ranks pairs by routed mass, so the same number of slots covers a different share of traffic depending on the card and the split. Read it as a strong trend with ±5 points of scatter, not as a lookup table.

The two-card 32k point used to sit at 4,636 slots and 92–94 %, which broke the trend and made this section's title look wrong. It was the --batch 3 figure again (§10): with one request at a time the same 32k run keeps 4,997 slots and hits 96.6 %, which is the best of the three two-card points, not the worst.

The obvious reaction to the 703-slot figure is: so leave it empty, the arena has the experts. Measured: no — a small cache is not a neutral state, it is the main thing decode is bound by.

Same 5070, same code prompt, only the context and the KV type change:

The log line for the worst case:

decode expert cache hit rate: 51.3% (65,707 hits / 127,965 lookups);
16,879 more read by the GPU over PCIe or from another GPU (11.7 % of all 144,844 routed)

Three things to read from that:

• A miss is not free. The 10.9–11.7 % figure beside it is the share of routed experts that had to be pulled across PCIe or from the other card. On a 28.7 GB/s link, 11 % of all expert traffic per token is what the missing 40 points of hit rate cost.
• The relationship is smooth, not a cliff. 2,054 slots → 72 %, 1,317 → 64 %, 703 → 53 %. Every slot you give back to the KV cache is taken from the hit rate almost linearly, which is why --kv q4_0 (+614 slots) buys +9 tok/s and a bigger window (−1,351 slots) costs −19.
• It also explains the two-card number. 4,361 slots give 96–98 %, and that is the whole reason two cards decode at 109.8 while the best single card manages 90.5.

So the answer is: yes, an almost-empty cache hurts, and it is the single largest factor in single-card decode speed. Keeping experts in the arena does not substitute for having them in VRAM — the arena removes disk reads, the cache removes PCIe transfers, and decode pays for the second one.

1.4. q4_0 changes sign with the context: it hurts at 256k on two cards, it is required at 256k on one 12 GB card

Why the prefill doubles — and it is not the arithmetic. The Hadamard rotation on every K/V access is a small cost either way. What actually moves is the prompt chunk, which the engine sizes from how much of the expert cache it may borrow, and a starved cache cannot lend much:

# int8: the cache is too small to lend a big chunk
expert cache auto: 1.86 GiB free ... -> 703 slots
prompt chunk auto: 2048 tokens, a 199-slot ring

# q4_0: halving the KV cache doubles the lendable slots, so the chunk triples
expert cache auto: 3.39 GiB free ... -> 1,317 slots
prompt chunk auto: 6144 tokens, a 199-slot ring

So q4_0 does not speed up the prompt path — it frees VRAM, which un-clamps the chunk from 2,048 to 6,144, which cuts the number of hand-offs on a 54k prompt from ~27 to ~9. The same clamp is described in §4, where the two-card profile was stuck at 7,168 until trim freed slots on the second card. On any single card at a large context, the chunk is the first thing to check before blaming the KV type.

Why the whole chunk does not "just fit" on the 5070. A prompt chunk is not free: while it is being read, every expert it needs must sit in VRAM, and those experts are exactly the slots borrowed from the expert cache. The engine will not lend them all — from generate.cpp:7631-7643 it lends at most 85 % of the slots (90 % when the copies are DMA from pinned RAM), and always keeps 128 slots untouched:

a 8192-token chunk fits when:  slots_for(8192) + 128 <= cache_slots()
                          and:  slots_for(8192) * 100 <= 85 * cache_slots()

On the 5070 at 262,144 with int8 the cache reaches 903 slots, so the budget is 0.85 × 903 − 128 = 639 slots — 8,192 tokens need far more. The scan walks the chunk down (8192 → 4096 → 2048 → …) until the loan fits, which is why it stops at 2,048. With q4_0 the cache gets 1,697 slots, the budget doubles, and the scan stops three steps later at 6,144. The cap is not a chunk-size limit and not a driver limit — it is the expert cache's size, deliberately kept partly free so decoding keeps hitting it.

Corollary: on a 12 GB card you cannot have both a full 8,192 chunk and a 262,144 window. The window takes VRAM first, the cache gets the rest, and the chunk is whatever the cache can lend. --kv q4_0 is the only knob that moves both ends — halving the KV buys back slots, which buys back chunk.

1.4b. The opt-in path to a bigger chunk: --prefill 16384

The scan is not hard-capped at 8,192: auto_ceiling starts at max(8192, min(--prefill auto_max, max_context)) (generate.cpp:7648), and 16,384 / 32,768 are documented as opt-in. The comment records the trade: on an RTX 5090 with IQ2_XS a 32K prompt read at 5,624 tok/s in 8,192-token chunks and 6,465 in one 32,768 chunk — "40K -> 18K experts streamed", i.e. a bigger chunk streams more experts per chunk.

That is the thing to watch here: more experts per chunk means more PCIe traffic, and both cards share 28.7 GB/s.

Measured, and it does nothing: raising the ceiling to --prefill auto:16384 on the 5070 at 262,144 with q4_0 left the chunk at 6,144 — the scan never reached the ceiling, because the slots ran out first. So the limit is the loan budget, not the 8,192 default.

STRATA_PREFILL_LEND_PCT (the 85 % above) is the other lever, and it is weak on this box:

+4 % chunk, and the measured prefill went down (2,041 → 1,997, inside spread but not a gain). Lending more slots means streaming more experts through PCIe during the prompt, and on a shared 28.7 GB/s link that eats the benefit. 85 % is the right default; leave it.

The conclusion for the 12 GB card stands: the chunk is bounded by the expert cache, and --kv q4_0 is the only knob that moves it meaningfully (2,048 → 6,144). To go past that you need slots, and slots on a 12 GB card at 262,144 do not exist — that is the hardware talking, not a flag.

The rule that falls out of these two measurements is not about the flag, it is about whose need is being starved:

On two cards the expert cache already has 4,361 slots and the KV cache is not squeezing anything, so the Hadamard rotation on every K/V access is pure cost (−2,1 % prefill). On one 12 GB card at the same window the KV cache has pushed the expert cache down to 703 slots, and halving the KV memory returns 614 slots — decode +7,1 %, prefill +85 %.

Practical: keep int8 when the slots are comfortable, switch to q4_0 when they are not. Do not carry either answer between configurations.

1.5. Smaller KV types buy slots but lose the prompt

The KV cache lives in VRAM on every card, so a smaller KV type should hand VRAM back to the expert cache — and decode here is bound by the expert cache. Measured on the code prompt at 262,144:

q4_0 does buy the most slots (+284) and a small decode gain on code, but the prompt loses 2.1 % — the Hadamard rotation on every K/V access costs more than the slots return. On prose the decode difference disappears (93.5 vs 92.9). k8v4 is worse on both. int8 stays, which matches the NVFP4 result where --kv fp16 cost 12.6 % decode for half the slots.

2. STRATA_STAGE_TRIM=1 needs a fixed K — and auto silently throws half the win away

Trim stops each card loading the dense weights of layers its stage never reads. --trim-stage-weights and STRATA_STAGE_TRIM=1 are the same switch (generate.cpp:3839 merges them with ||).

Measured alone, on the same K=27:

The trap: with layer_split: auto the first card does not get trimmed. It builds its arena before the split search runs (generate.cpp:3845), so only the later cards trim. Trim therefore only pays off together with a fixed K — and the two together are worth more than either alone:

Slots went down from 6,147 to 4,361 and decode went up: the pinned arena matters more than a few hundred cache slots, and the trim is what frees the VRAM that makes the K=27 placement affordable.

3. Split-point sweep (trim on, context 262,144, prose prompt)

The decode and prompt optima do not coincide. Decode peaks at K=27 (+7.8 % over auto), the prompt at K=25 (+1.3 %). K=27 costs 4.9 % of prompt speed against K=24. I kept K=27 because decode is paid per token while the prompt is paid once per request; anyone with long contexts and short answers should prefer K=25.

This sweep was taken at STRATA_ARENA_PIN_GIB=40. Re-running the endpoint at 52 (the registration path of §1) lifts the same K=27 to 2,896 prefill and 109.8 decode on code — so the K choice is independent of the pin value, but the absolute numbers in this table are the 40-pin ones.

K=27 and K=28 differ by 0.5 tok/s, which is inside the spread — treat them as equal.

4. The prompt chunk stops being clamped

With the default setup the engine cut the prompt chunk on IQ3_S:

WARNING: prompt chunk 7168 tokens, not 8192: CUDA1 (RTX 5070) has 1770 expert-cache slots, and a 8192-token chunk borrows 1692 of them

After trim + K=27 the second card holds fewer dense weights and lends more slots (2,747), so prompt chunk auto: 8192 tokens — the clamp is gone, which is where the prefill gain comes from.

5. Three expert modes, only one is right here

The pack has an experts.bin. Its mere presence makes setup.py write --resident-experts, so on this profile the file is renamed to experts.bin.off to keep the arena:

6. On a real workload the numbers are higher than any prose benchmark

Same flags, two prompts, 5 runs each, warm-up discarded:

Code is simply more predictable, so the MTP drafter lands more often: +13…17 % decode with identical flags. Sweeping the speculator on the code prompt shows the default is already the optimum:

7. One card versus two — and the two cards swap roles

All three points at context 32768 (262,144 does not fit one card), same code prompt:

The 5070 is faster at prefill (+34 %: 48 SMs against 36 at nearly equal clocks, and the prompt path is compute-bound). The 5060 Ti is faster at decode (+4.7 %) despite fewer SMs, because it holds 79 % more expert-cache slots (3,685 vs 2,054) — decode with MTP is bound by cache hits, not by compute. The two-card row is 30 % above the faster single card at the same window; the growth comes almost entirely from slots (2,054 → 4,997 is ×2.43, decode ×1.36), not from the second card's compute. On two cards at 262,144 the same prompt decodes at 109.8, i.e. the larger context itself costs 4.8 % — the KV cache takes VRAM the expert cache would otherwise hold.

Every row here is one request at a time. Without pinning parallel: 1 the app would put 3 slots on the two-card run (see §10) and the comparison would mix two configurations.

Which card to put in a second PC? The 5060 Ti, not the 5070 — at 262,144 it keeps 2,333 expert-cache slots against the 5070's 703, decodes 78.2 against 71.7, and needs no --kv q4_0 to stay usable in prefill (1,796 against 1,101). The 5070 only wins on prefill at small contexts, where its 48 SMs matter.

Device indices, for anyone reproducing this: nvidia-smi reports 0 = RTX 5070 (12,227 MiB), 1 = RTX 5060 Ti (16,311 MiB), and the engine sets CUDA_DEVICE_ORDER=PCI_BUS_ID (generate.cpp:2421), so its numbering matches. "gpu": [1, 0] is an order, not a filter: the split stages are taken in that order, so CUDA1 is the 5070 and the last entry is the Main card holding the head.

8. Single-card gotchas worth knowing

• A layer split on one GPU (--split-device 0) is possible, but then the engine refuses conversation parking outright: conversation parking does not support a layer split on one GPU (--split-device 0) (generate.cpp:3041). Drop --layer-split instead.
• STRATA_STAGE_TRIM=1 prints --trim-stage-weights needs a layer split across GPUs (ignored) on one card.
• STRATA_ARENA_PIN_GIB only matters for a multi-GPU run; a single card registers the whole arena regardless.
• A single card at 262,144 allocates the KV cache first and the expert cache from what is left, so --expert-cache auto can shrink to 703 slots (12 GB card) — at that point --kv q4_0 is the knob that matters, not the cache tunables.
• Check prompt chunk auto: before trusting any single-card prefill number. The engine sizes it from the slots the cache can lend, so a starved cache silently clamps it to 2,048 and reads a 54k prompt in ~27 hand-offs instead of ~9. On the 12 GB card that alone is 1,101 against 2,041 tok/s — a bigger effect than the q4_0 flag it gets attributed to.
• The clamp is not reported as a warning on a single card (it is, on a split: prompt chunk 7168 tokens, not 8192), so on a one-card run it must be read from the line itself, not from a WARNING.

9. Keeping experts in RAM does not free VRAM

The natural idea for a tight card: move the experts to RAM. --resident-experts does exactly what it says — resident RAM mode: 37.34 GiB of experts in RAM (page-locked) ... 0 blob reads from the file — and VRAM usage does not change at all:

--expert-cache auto sizes itself from free VRAM, so the moment the disk reads stop, the cache fills the space instead. The only way to actually leave VRAM unused is to cap the cache by hand — and that is expensive: --expert-cache 1200 on top of --resident-experts freed 5,214 MiB on the second card and moved 41.72 GiB of experts into RAM, but decode fell to 77.8 (−29 %). Each slot handed to free VRAM is taken from decode almost one for one.

So on this rig the answer is: keep --expert-cache auto, keep --vram-reserve-mib 0, and treat 0 MiB of VRAM free as the normal state. The LOW: requests may stall warning is constant here and cost nothing measurable — 109.8 tok/s held across runs.

10. The app adds --batch itself on small contexts — and it costs up to 18 % of decode

This one cost me a wrong first draft of the at-a-glance table, so it is worth writing down as a trap rather than as a tuning result.

--batch is not in this profile. The desktop host fills it in on its own: host/server/native-parallel.mjs chooseParallel() runs on every load, its PARALLEL_POLICY is 'fit', and it takes as many concurrent request slots as fit inside CACHE_LOSS_SHARE (20 %) of the solo expert cache, aiming for DEFAULT_SLOTS = 4. On this machine that resolves to:

The engine says it plainly at start: --batch 3: the slot sessions take 0.89 GiB of VRAM on CUDA0 that the expert cache would otherwise hold, and --batch-groups auto: 1 group of 3 slots. A 3-slot run also reports one token figure per slot, so the CSV of such a session looks slower in a way that has nothing to do with the flags being compared.

The fix is one key. parallel: 1 is a number a person chose, so chooseParallel() returns it untouched instead of computing one — and the numbers above are exactly that A/B: same cards, same K, same --kv int8, same STRATA_ARENA_PIN_GIB=52, only the key changed.

# iq3s-128gb/set-parallel.py 1     -> one request at a time, the benchmark baseline
# iq3s-128gb/set-parallel.py auto  -> remove the key, the app decides again

Two consequences worth stating:

• Every row in this report is a one-request row. For 32k and 64k that required pinning parallel: 1; the default profile is left with the key removed, because concurrency is a feature, not a regression — a lone chat is not slower for having slots, it is the benchmark that must not have them.
• The direction of the old numbers. My first series had 32k at 99.5 and 64k at 94.0, which read as "decode falls as the window grows, then recovers at 128k". That shape was an artefact of the slot count: 3 slots at 32k, 2 at 64k, none from 128k. Measured one request at a time the curve is monotone — 115.3, 114.7, 112.5, 111.6, 109.8 — and nothing "recovers" at 128k. Cross-checking a table against the --batch line in the log is now step one for me, not step last.

Negative results, each rejected for a stated reason

--pipeline-windows 1 (works — two verifiers per stage, 2 stages overlap across windows — but the prompt is already read in one continuous stream: prefill 2,668.1 vs 2,664.7, decode 92.1 vs 92.9, and the second verifier takes 160 MiB out of the expert cache: 4,361 → 4,298 slots); --pipeline-windows 2 (decode 87.6, −14 % slots: the draft chain sits on the last card here and shares it with its window, exactly the case generate.cpp:4331 warns about); 2 + STRATA_MTP_STAGE0=1 (decode 74.5: the draft layer on the first card costs more VRAM than the overlap returns); STRATA_PREFILL_NVFP4=w4a4 (the mode only selects a kernel for GGML_TYPE_NVFP4, moe_mmq.cu:356/:481; measured 2,690.9 vs 2,664.7 — noise, flag removed); --max-context 204800 at K=27 (decode 91.1, prefill 2,735.7 — not faster than 262,144 here, so the smaller window buys nothing); --kv q4_0 and --kv k8v4 on the two-card 262,144 profile (§1.5); STRATA_ARENA_PIN_GIB=0 (§1); --spec 3, --spec 5, --spec-min-p 0.3, --spec-min-p 0.7 (§6 — the shipped 4 / 0.5 wins on code too); --resident-experts (§9 — no VRAM freed, 6.1 % slower); --resident-experts + --expert-cache 1200 (§9 — frees 5.2 GiB, costs 29 % decode).

Note on --kv q4_0: it is in this list for the two-card profile only. On a single 12 GB card at 262,144 it is required, not rejected — see §1.4. Same flag, opposite verdict, because the question is whether the KV cache is squeezing the expert cache out.

What was deliberately not tried

--native SHARD1 from strata serve --help is not a separate switch: --native takes the model path (generate.cpp:2536, o.native_preset = next("--native")), and on this profile it already points at the model. Overwriting it to enable the "all native paths at once" bundle would risk the load for an unproven gain, so it stays untested — worth a separate run on a copied profile rather than on the working one.

Limitations, stated plainly

Numbers come from my own fixed prompts and scripts, not from strata bench run, so they are not standard-protocol results and should not be compared row-for-row with them. Decode: 5–8 runs per point via the API, warm-up discarded, the reported figure taken from the engine line N generated in … ms (… tok/s). Prefill: unique text (a Guid in every paragraph, reused = 0), figure from prompt N tokens = 0 reused + N read in … ms. The model was fully unloaded and reloaded between configurations — layer_split and env are read only at engine start.

No TTFT, no needle/recall, no test-run coding tasks, no tool calls, no concurrency, no power capture. Quality was checked for encoding damage only (0 U+FFFD, 0 control characters), not for answer quality.

One correction I had to make to my own draft: an earlier version of this report stated that a single card tops out at 128k context. That was an assumption, not a measurement, and it is wrong — 262,144 loads and runs on both cards (§ the at-a-glance table). Anything in this report that reads like a limit is now a measured point; where I state a limit without a row, treat it as unverified.

Two things are still unmeasured and would change the single-card picture: k8v4 at 262,144 on the 12 GB card (it might land between the 703 and 1,317 slot figures), and --kv q4_0 at 128k on the 16 GB card (3,105 slots may not be the ceiling there).

Spread inside one configuration is 6–9 tok/s, wider than the gap between neighbouring values. Only the peaks and the large steps are meaningful: K=27 over auto (6.8 tok/s), trim (8.0 tok/s) and the pin threshold (2,682–2,704 → 2,877–2,917 tok/s prefill) are solid; K=27 vs K=28 is not, and the decode part of the pin gain (1–4 %) is at the edge of noise. Decode figures for the single-card runs come from 5 runs each, so treat the 4.7 % gap between the two cards as indicative rather than settled.

One correction the spread does not explain: the first 32k and 64k two-card figures (99.5 and 94.0) were not noise but --batch (§10). Re-measured one request at a time they are 115.3 and 114.7, and four independent runs of each agree within 3 tok/s. That is the largest single correction in this report, and it came from reading the engine's start-up lines rather than from trusting the CSV.

One caveat on the draft vocabulary: the Cyrillic subset was already active on this box (draft head over 58,963 tokens, i.e. draft_vocab_cyrillic.bin), and it is not selected by any config key — the file is shared and a plain copy. Re-running model setup overwrites it.

Conclusion: the configuration to keep, and the four rules behind it

The profile I run, and the numbers it gives on this machine:

// install/strata-iq3_s.json — the parts that matter
{
  "args":        ["--layer-split", "27", "--max-context", "262144", "--kv", "int8",
                  "--spec", "4", "--spec-min-p", "0.5", "--mtp", "...\\mtp\\rt",
                  "--vision", "--expert-cache", "auto", "--prefill", "auto",
                  "--host-core", "sibling", "--vram-reserve-mib", "0"],
  "gpu":         [1, 0],                 // CUDA0 = 5060 Ti, Main (head + drafter) = 5070
  "layer_split": "27",
  "env": {
    "STRATA_ARENA_PIN_GIB": "52",        // whole arena in one pin, +7 % prefill
    "STRATA_STAGE_TRIM":   "1",          // +629 slots, needs a fixed K
    "STRATA_ARENA_READERS": "32",
    "STRATA_ARENA_CHUNK_MIB": "16"
  }
  // no "parallel": the app's own slot policy applies (see §10)
}

Four rules, in the order they paid off, each one measured as an A/B on this box and nothing else changed:

1. Stay in the arena. experts.bin renamed to .off keeps the engine streaming through pinned slices. --resident-experts removes disk reads and looks safer, and costs 6 % decode while freeing no VRAM at all (§9).
2. Pin the whole arena, not 8 GiB of it. The 8 GiB cap is the engine's default, not a WDDM limit: STRATA_ARENA_PIN_GIB=52 turns 6 slices pinned (7 GiB) into one cudaHostRegister PORTABLE ok and buys ~7 % prefill. Past ~54 GiB it reverses (§1).
3. Fix K and trim. STRATA_STAGE_TRIM needs an explicit --layer-split; with auto the engine trims only the later cards and half the gain disappears (§2). K=27 is the decode optimum, K=25 the prefill one, and they do not coincide (§3).
4. Benchmark one request at a time. This is not an engine flag at all — it is the host adding --batch 3 / --batch 2 on small contexts and taking up to 0.89 GiB of expert cache with it. Without pinning it, the same machine reports 100.7 where it really does 115.3 (§10).

What I would not do on this rig, each rejected by measurement rather than by reading: --kv q4_0 at 262k on two cards (helps at 768k and on a starved single card, hurts here), --kv fp16 or k8v4, --pipeline-windows, --prompt-cache-spread, --expert-cache N by hand, --vram-reserve-mib above zero on display-less cards, --resident-budget-gib, STRATA_PREFILL_NVFP4=w4a4, STRATA_EXCHANGE_ROTATE, STRATA_PIN_BY_NEED under a layer split, and any attempt to buy VRAM by shrinking the cache.

The one-line summary: on two mismatched Blackwell cards this model reaches 115 tok/s decode and 2.9k tok/s prefill at a full 262k window, and the 1M window it can also load is decode-only — prefill there is 1,613 tok/s against 2,882, because the prompt chunk collapses from 8,192 to 2,816 while the slot count falls to 1,091. Past 262k you are buying window with prompt speed, roughly linearly.

What is still open, stated so nobody reads more into the tables than is there: the decode spread inside one configuration is 6–9 tok/s, wider than several of the gaps quoted, so the ranking of neighbouring points (K=27 vs K=28, and the 1–4 % decode part of the pin gain) is not settled; k8v4 at 262k on the 12 GB card and q4_0 at 128k on the 16 GB card are unmeasured; and no TTFT, concurrency, recall or answer-quality test was run — only encoding damage was checked.