Skip to content

[Benchmark] RTX 4090 48 GB + Qwen3.8-Flash-Next IQ3_S: long-output decode 158-263 tok/s, driven by MTP acceptance #4768

Open
Open
[Benchmark] RTX 4090 48 GB + Qwen3.8-Flash-Next IQ3_S: long-output decode 158-263 tok/s, driven by MTP acceptance#4768

Description

@wangminle
wangminle
opened on Oct 31, 2026

Problem

tools/strata_bench.py never exercises long generation. The caps are 100 new tokens for short chat, 24 for the 4K/16K/32K prompt tests and 64 per agent turn; --long only makes the prompt longer (32K). Most of the work people do with this model — writing a module, emitting a large JSON document, reasoning for thousands of tokens — happens above those caps, and the decode speed there is not the same as in short chat.

I added a small long-output bench (four cases, 1.9K–7K generated tokens, with a check where the answer is verifiable) and ran it 3× per case against two launch configurations on one RTX 4090 48 GB. The result is that long-output decode ranges from 158 to 263 tok/s and follows MTP draft acceptance almost exactly, while the same server reports 165–168 tok/s for 100-token chat.

Hardware and software

  • GPU: NVIDIA GeForce RTX 4090, 49,140 MiB VRAM, driver 595.91.07, PCIe gen4 x16, power limit 450 W
  • CPU: AMD EPYC 9B14, 96 cores / 192 threads; RAM 282.8 GiB; NVMe storage
  • OS: Linux 7.0.0-42-generic; CUDA 13.2 (driver)
  • Strata Terminal 0.2.7 (build 0.1.53), engine 0.1.52, Linux CUDA build
  • Model: Qwen3.8-Flash-Next, ISTA-DASLab GSQ-RCO IQ3_S (two GGUF shards, 84.5 GB), MTP draft layer
  • Context 131,072 tokens, KV int8, kv-resident 32768, temperature 0 throughout

Configurations

Both were launched from the release Terminal against the same downloaded model, on the same machine, within the same hour. Paths replaced with <data-dir> / <feed-dir>.

A — stock launch (what strata launch --family qwen --model iq3_s picked by itself):

strata launch --family qwen --model iq3_s --context 128k --vision gpu --yes --no-tui

Engine reported: parallel_slots=4, spec=6, mtp_max=4, spec_min_p=0.50, pcie_frac=0.55, prefill_cap=8192, expert cache 18,615 slots / 36,305 MiB (35.45 GiB), 435 MiB VRAM free.

B — tuned launch (text only):

strata launch --family qwen --model iq3_s --context 128k \
  --vision off --parallel 1 \
  --set spec=6 --set spec-min-p=0.70 --set pcie-frac=0.35 \
  --set vram-reserve-mib=1024 --set prefill=auto:32768 \
  --set STRATA_HC_Q8=1 \
  --yes --no-tui

Engine reported: parallel_slots=1, spec=8, mtp_max=6, spec_min_p=0.70, pcie_frac=0.35, prefill_cap=32768, expert cache 20,936 slots / 40,853 MiB (39.90 GiB), 746 MiB VRAM free.

Why these settings, following the 48 GB reports in this repo: turning vision off frees the vision reader's VRAM for the expert cache (I did not measure the split); one parallel slot avoids the batch slots the stock launch reserved (4 slots, ~2.9 GiB) and keeps VRAM headroom above the low-VRAM warning; a higher reserve (1024 MiB) leaves room for the ring buffer; a 32K prefill cap removes the 8K chunking seen at the start of long prompts; the deeper draft (spec=6, engine reports mtp_max=6) and a stricter spec-min-p were chosen to push draft acceptance. STRATA_HC_Q8=1 was enabled as an opt-in; I did not isolate its effect. B has no image input. For vision, use A.

Standard suite (python tools/strata_bench.py --long, 3 runs, medians with ranges)

Test A decode tok/s B decode tok/s A prompt tok/s B prompt tok/s
short chat, 100 tok 164.9 (124.5–207.7) 168.3 (133.7–247.9) not measured not measured
4K prompt, 24 tok not measured not measured 3234.3 (3210.8–3290.2) 3130.3 (3123.5–3175.8)
16K prompt, 24 tok not measured not measured 4372.1 (4345.9–4461.9) 4202.1 (4160.4–4268.5)
32K prompt, 24 tok not measured not measured 4510.3 (4410.3–4540.9) 4439.4 (4336.7–4466.9)
agent turn, 64 tok 166.1 (150.9–181.8) 164.5 (137.7–191.9) — —

Chat +2.1 %, agent −1.0 %, 4K prompt −3.2 %, 16K −3.9 %, 32K −1.6 %. On the standard suite the tuned launch is roughly neutral for decode and slightly slower for prompt processing.

Long-output bench

Method: streaming /v1/chat/completions, temperature=0, reasoning_effort=none except the thinking case; each case 3 runs. Speeds are the engine's timings.predicted_per_second; TTFT is measured by the client. Draft acceptance is accepted/offered from timings, reported per request then medianed.

Case (max_tokens) Config Generated tokens Decode tok/s Draft acceptance TTFT s Total s Check
code module + tests (4096) A 2360 206.6 (195.9–208.3) 81.5 % (81.2–82.9) 0.19 11.7 3/3
B 2431 227.1 (227.0–228.2) 82.7 % (82.2–82.8) 0.16 10.8 2/3
math, no thinking (3072) A 3072 179.6 (176.4–185.5) 68.8 % (67.5–71.3) 0.22 17.3 0/3
B 3072 183.5 (181.7–187.9) 72.5 % (71.4–73.0) 0.06 17.0 0/3
math, thinking (8192) A 6807 163.2 (162.1–163.4) 69.8 % (69.5–69.8) 0.20 41.8 0/3
B 6860 158.4 (156.9–158.9) 76.4 % (75.6–76.6) 0.07 43.6 0/3
400-line log → JSON (4096) A 2865 227.4 (216.1–229.2) 95.1 % (93.9–95.5) 0.36 13.0 3/3
B 2845 263.2 (260.5–265.6) 96.6 % (96.0–96.7) 0.13 10.9 3/3

Notes:

  • The JSON case has a 16,277-token prompt. Its first run in each set shows the cold TTFT (3.5–3.8 s); the second and third hit the conversation cache (16,270 reused tokens) and start in 0.08–0.36 s.
  • The code and math cases have 124–245-token prompts, so their TTFT is mostly model overhead (0.06–0.60 s).
  • Case definitions: (1) code — a thread-safe TTLCache with O(1) operations, TTL, purge(), 8+ unittest methods; the check extracts the code block, runs it and its own unittest suite, and requires exit 0 and ≥8 tests. (2) math — count 6-digit integers with distinct nonzero digits divisible by 7, answer on a FINAL: line; expected 8544, computed by brute force over the 60,480 candidates. (3) same with thinking on. (4) JSON — 400 deterministic synthetic log lines containing exactly 40 ERROR ... incident=INC-xxxx lines; the check parses the JSON, requires all 40 IDs and a summary over 20 words.

What the numbers say

  1. Draft acceptance, not token count, sets long-output speed. The two structured cases (code 82.7 %, JSON 96.6 %) decode at 227–263 tok/s; math at 68–76 % acceptance decodes at 158–184 tok/s; short chat sits at 165–168 tok/s. The JSON case is about 56 % faster than short chat on the same server with the same model state.
  2. Long structured output is where the tuned launch pays. +15.7 % JSON, +9.9 % code, +2.2 % math-without-thinking over stock, with roughly neutral short chat and −1 to −4 % prompt throughput.
  3. The 6.8K-token thinking case is the exception: B raises acceptance from 69.8 % to 76.4 % but decodes 3 % slower than A (158.4 vs 163.2 tok/s). I did not isolate the cause; my guess is longer per-sequence KV traffic and more draft tokens verified per step, but that is unconfirmed.
  4. Long outputs also expose quality failures that a 100-token suite cannot: the math problem fails in both configurations (see below), and one tuned code run generates a module whose own tests fail.

Failed checks (kept separate from the speed results)

  • Math, 0/3 in both configurations. With thinking off, all six requests hit the 3072 cap before printing FINAL: — the problem needs more room than the prompt's "under 2,500 tokens" asks for. With thinking on, the model always finishes but gives 8640 (all three B runs, two A runs) or 8537 (one A run) instead of 8544. The failure is consistent, not noise.
  • Code, 2/3 in B, 3/3 in A. B run 0 produced a module that fails its own test_negative_ttl_does_not_store (exit 1 on both the direct run and unittest); the other five code runs pass.
  • No request errored: all 24 long-output requests returned finish_reason stop or length with engine timings.

Not tested / limits

  • One GPU, one model file, one machine, one day. No multi-GPU, no concurrency/batch throughput, no other quantization, no other model family.
  • B is text-only (vision off). A was measured with vision on but no image was sent, so this is not a vision benchmark.
  • Background load was not recorded; the machine was otherwise idle.
  • The standard suite for A used the launch's automatic parallel=4; I did not run A with parallel=1 alone, so the prompt-throughput difference mixes several changes.
  • The checks are functional smoke tests (code executes, JSON parses, a number matches). They are not a quality evaluation, and generated code is executed in a temp directory without a sandbox review.
  • spec=6 is the launch flag; the engine reports spec=8, mtp_max=6 for B and spec=6, mtp_max=4 for A, so the flag and the reported field do not use the same scale.

Proposal

Add an opt-in long-output mode to tools/strata_bench.py — four cases like the above, 3K–8K caps, engine timings, draft acceptance, and the code/JSON/math checks — so this workload is measured the same way on every GPU report. The script and both results.json sets are ready; I can open a PR that adds bench/long_output_bench.py unchanged (no default path touched) if that fits the project. Happy to re-run on the final engine build or with any configuration you want compared.

Related: #3265 (expert lookahead on RTX 5070) is the other recent 48 GB-class decode experiment; this report is about long generation and launch flags, not expert lookahead.

Activity

  1. udqe commented on Oct 31, 2026

    @udqe
    udqe
    on Oct 31, 2026

    A second box for the acceptance story: Windows, 2x RTX 4090 24 GB, CUDA, engine 0.1.54, our production config (the one in #4573), UD-Q4_K_XL native pack, T=0, 21 requests this morning in two blocks. Source for every number: the engine's own /metrics per-request fields - which carry your acceptance metric on CUDA too: drafts_offered, drafts_accepted, decode_tok_s, hit_rate, first_engine_token_s.

    case runs decode tok/s (range) acceptance check
    short chat, 100 tok 3 233.6 - 261.1 86.6 - 94.7 % see note 3
    math, 3072 cap 13 133.7 - 148.1 76.9 - 81.7 % 0/13
    JSON from 400 log lines (~15.2K prompt) 7 196.7 - 213.1 90.0 - 92.4 % 7/7

    Four things this adds rather than repeats:

    1. The monotone holds across content types within one session. Between our own chat traffic (751 tok, 82.6 % acceptance, 179.8 tok/s) and these cases, decode tracks acceptance with no visible second factor - the two-card version of your point 1, where expert-cache misses ride on top.

    2. The math failure is quantization-independent and bigger than 0/3. You saw 8640/8537 (IQ3_S). We raised to 13 runs (UD-Q4_K_XL): 0 correct, nine distinct wrong answers - 8640 x3, 7200 x2, then one each of 8537, 8532, 8376, 8748, 7560, 8628, 7776 - plus one length-cap miss. The expected value was brute-forced independently here: 8544, as you said. Two quantizations, two machines, one consistent arithmetic weakness at ~3K-token outputs, not a quant artifact. Worth detaching from this thread's speed conclusion.

    3. A short-output failure your suite would only catch by luck. Second block, same session, same prompt as the first block's three good short runs: message.content came back EMPTY 3/3 (finish length, 100-200 tokens all consumed by reasoning_content - one debug run captured 801 chars of reasoning and zero answer). The 100-token budget fits either the reasoning or the answer, not both. You set reasoning_effort=none in your suite; we did not, and that is exactly the difference - so pin it in the proposed long-output mode, because a 100-token short run on this default silently returns nothing at all.

    4. Your flag-vs-report scale note reproduces on CUDA/Windows. Our launch passes --spec 4; /metrics reports "spec":6, "mtp_max":4. Different scales on your Linux launch and our CUDA one - whatever the long-output mode prints, name the scale.

    One axis caution: same box, same hour, your JSON-style case reads ~210 tok/s while our standing cold-prompt bench (150K tokens, 0 reused, arena deliberately cold) reads 95-110. Acceptance and cache-state both move this number; pin the reuse state the way the issue already pins acceptance.

    Limits kept honest: one session, one day, one box; prompts reworded, ours not yours; the code-execution case skipped by policy (we do not run model-generated code here); non-streaming client (TTFT = the engine's own field only); math answers counted from the FINAL: line where present. This box carries the +-10 % day-to-day decode spread we documented in #4610 - deltas under ~10 % are noise; the acceptance-decode gap (91 % vs 78 %, ~1.5x) is far above it.

    Happy to re-run any configuration. +1 for bench/long_output_bench.py landing as-is, with the fields wired to /metrics instead of the response body - fewer build-dependent JSON paths.

    (AI-assisted again on our side - the model served by this repo's engine ran the suite and read its own metrics; all figures local.)

  2. wangminle commented on Oct 31, 2026

    @wangminle
    wangminle
    on Oct 31, 2026
    Author

    Follow-up: pass-only rerun (8 runs per case)

    I re-ran the two cases whose checks can pass, 8 runs each on both launch configurations, and summarised only the requests whose check passed. The table in the report above includes every request that returned, including code runs whose own tests failed; this one separates the two.

    Same machine, same day, same commands as the report; the server was restarted between the stock and tuned sets. Medians with ranges over passing requests.

    Case Config Checks passed Decode tok/s (passing) Generated tokens TTFT s Total s Draft acceptance
    code module + unittest stock 8/8 210.8 (198.7–215.7) 2380 0.205 11.43 82.9 %
    tuned 4/8 231.2 (229.7–238.6) 2450 0.191 11.05 81.3 %
    400-line log → JSON stock 8/8 228.1 (210.9–232.9) 2870 0.358 12.94 93.6 %
    tuned 8/8 274.5 (266.9–280.1) 2863 0.200 10.71 96.0 %
    • JSON: tuned +20.3 % decode (228.1 → 274.5 tok/s), total time −17.2 %.
    • Code: tuned +9.7 % decode (210.8 → 231.2 tok/s), total time −3.3 %.
    • The JSON prompt is 16,277 tokens. Its first run is cold (TTFT 3.5–3.8 s); the other runs reuse 16,270 tokens from the conversation cache, so the median TTFT above is the warm value in both configurations.
    • The tuned code median is over the 4 passing requests. The tuned code set's 4 failing runs are excluded from speed here.
    • These numbers are somewhat above the first 3-run set (tuned JSON was 263.2 tok/s there). That is session-to-session variation on the same machine; the 8-run medians are the more stable estimate, and the relative result is the same direction.

    Pass rate differs on the code case

    The checks are the same as in the report: extract the Python block, run it, run its unittest suite, require exit 0 and at least 8 test methods.

    • stock: 8/8 passed. tuned: 4/8 passed.
    • Failure modes in the 4 tuned runs: 3 use unittest.TestCase / unittest.main() but omit import unittest (NameError on both the direct run and python -m unittest); 1 has test_default_ttl and test_ttl_expiry failing with AssertionError: 1 is not None (the default TTL / expiry behaviour is wrong).
    • A two-sided Fisher exact test on this one rerun (8/8 vs 4/8) gives p ≈ 0.011. I am reporting it as an observation, not a cause: the tuned launch also differs in vision, parallel slots, draft depth, prefill cap and STRATA_HC_Q8, and I did not isolate which of them, if any, is responsible. A larger paired run would be needed to call it a property of the configuration.
    • This pass-rate difference is a generation-quality signal, separate from the speed measurement. It does not affect the pass-only speed numbers, but it does mean the tuned code median rests on 4 samples.

    What this changes in the report

    • The "acceptance sets long-output speed" point holds within a configuration, across cases. Between configurations it is not sufficient: tuned is 9.7 % faster on the code case with lower acceptance (81.3 % vs 82.9 %), consistent with the larger resident expert cache (39.9 GiB vs 35.5 GiB), though I did not log cache hit rates during this rerun.
    • The math cases still have no passing samples under either configuration (0/3 each in the report), so there is no pass-only speed for them; they were not rerun in this pass.

    Raw results.json for both sets is available if useful. bench/long_output_bench.py now records both all-run and *_passing statistics and accepts --cases for a focused rerun.

  3. wangminle commented on Oct 31, 2026

    @wangminle
    wangminle
    on Oct 31, 2026
    Author

    Same rig, same tuned launch as this report, another workload: I kept the engine timings of a prefill-heavy real task (static review of a 330-file repository) and posted the record in #4813.

    • 162 sequential requests: 3,508,409 prompt tokens + 125,127 generated in ~25 min of request time.
    • Cold prefill weighted 4,773.2 tok/s (fast pass, no thinking) and 4,485.1 tok/s (deep pass, thinking); per-request medians 4,774.0 / 4,665.5 tok/s over 3.1K–24.3K-token prompts.
    • Decode 171.0 / 161.6 tok/s (weighted), TTFT median 4.63 / 4.07 s, MTP acceptance 73.4 % / 80.3 %.
    • The deep pass had higher acceptance but a lower decode median (161.8 vs 167.4 tok/s). In this workload acceptance alone did not set decode speed; I did not isolate the cause.

    Not the standard protocol (no fixed prompts, no repeats, one run per pass); the limits are in #4813.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions