NInfer-all

A C++/CUDA inference engine for the Qwen3.5 model family (Qwen3.6 and Qwen3.8 27B, Qwen3.6-35B-A3B, their fine-tunes and Ternary Bonsai 2), Qwen3.8-Flash-Next, Gemma 4 26B-A4B, Recursive's Flint and DeepSeek-V4.1-Flash, on NVIDIA GPUs from Volta to consumer Blackwell, from 8 GB of memory. It serves the OpenAI, Anthropic and llama.cpp APIs from one GPU, or from up to eight as pipeline stages on Linux. The same chat model answers typed decisions: choice, yes/no and score questions about a state, each option with its probability, through TypeSafe's System One and OpenAI's Decisions APIs.

What's New · Quick start · Supported GPUs · Models · Highlights · Features · Docker · Building · Documentation

NInfer-all continues NInfer through the NInfer-3090 line and carries the work of several forks; CONTRIBUTORS.md credits each of them.

Note

Support GPU testing. Choose a custom amount on GitHub Sponsors to help cover GPU rentals for correctness checks, kernel tuning, and benchmarks. This is a voluntary contribution without sponsor benefits. For compute credits, GPU access, or individual sponsorship terms, contact the maintainer directly.

What's New

• 2026-11-03: llama.cpp control vectors and direction LoRA adapters for Qwen3.5-core models, with per-request adapter scales.
• 2026-11-02: Tensor parallelism selects exchange routes by size from startup timings, including peer writes.
• 2026-10-31: Automatic tensor splits adjust GPU shares using free memory after the first load.
• 2026-10-30: Bundled media libraries, pre-Blackwell NVFP4, Qwen3.5 FP32 logits, attention kernel preload, and prefill pacing.
• 2026-10-29: Faster tensor-parallel prefill with reduce-scatter and native NVFP4/FP8 on RTX 50.
• 2026-10-27: Faster decode without peer access, full Flash-Next tensor parallelism, and invariant attention for Qwen3.5.
• 2026-10-27: The DeepSeek-V4.1-Flash STQ drafter uses half the VRAM of Unsloth's drafter.
• 2026-10-26: Faster Flash-Next prefill and decode with disk experts.
• 2026-10-26: Tensor parallelism for all dense weight formats and DFlash, plus expert parallelism for all MoE families.
• 2026-10-26: DeepSeek-V4.1-Flash gains CPU-assisted expert execution, DSpark drafting, and image input.
• 2026-10-25: DeepSeek-V4.1-Flash support with IQ1_M GGUF and host or disk experts.
• 2026-10-25: --tensor-parallel for dense Qwen3.5-core models.
• 2026-10-24: Flint decodes 348 tok/s on an RTX 5090 and 210 on an RTX 4090 with host experts.
• 2026-10-23: Flint from Unsloth's UD-Q4_K_XL GGUF.
• 2026-10-23: Native releases for Windows and Debian 13 with CUDA 12.9 and 13.4.
• 2026-10-22: Typed decisions: /v1/decisions and /v1/systemone.
• 2026-10-22: YaRN for every model family.
• 2026-10-20: CPU threads compute a share of host experts beside the GPU.
• 2026-10-18: The hybrid prefix cache is the server default.
• 2026-10-17: Pipeline stages across GPU architectures in one build.
• 2026-10-16: Device profiles for 36 more GPUs; Gemma 4 and Qwen MoE experts in host memory or on disk.
• 2026-10-14: Turing (sm_75): the TITAN RTX and RTX 20 series.
• 2026-10-12: Volta (sm_70): the Tesla V100.
• 2026-10-11: Gemma 4 26B-A4B with MTP, images and video.
• 2026-10-06: Qwen3.8-Flash-Next, experts on the GPU, in host memory or on disk.
• 2026-09-26: NInfer-all and Qwen3.8-27B GSQ-RCO IQ3_S.
• 2026-09-23: Ternary Bonsai 2 27B on one RTX 3090.

Quick start

Download an artifact (all models) and serve it from the Docker image. The server speaks the OpenAI and Anthropic APIs on http://localhost:8080/v1, answers typed decisions there, and each card picks up its device profile on its own.

hf download WaveCut/Ternary-Bonsai-2-27B-NInfer-v3 Ternary-Bonsai-2-27B-ninfer-v3.ninfer --local-dir models

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all:cuda13.4 serve /models/Ternary-Bonsai-2-27B-ninfer-v3.ninfer \
  --model-id bonsai2-27b --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 \
  --gdn-state-fp16 --spec dflash2 --draft-tokens 5

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"bonsai2-27b","messages":[{"role":"user","content":"Hello"}]}'

curl http://localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"ticket": "My card was charged twice for one order."},
  "questions": {
    "team": {"type": "choice", "instructions": "Which team should handle this?",
             "criteria": {"billing": "payments and refunds", "shipping": "deliveries"}},
    "urgent": {"type": "noul", "instructions": "Does the customer need a reply within the hour?"}}}'

The second request is a typed decision: each answer comes with the probability of every option, read from the model's first-token log probabilities of the labels.

Flags for every model and memory size, from an 8 GB card to the largest context: launch recipes.

Without Docker — the same flags on a build

ninfer-serve models/Ternary-Bonsai-2-27B-ninfer-v3.ninfer --model-id bonsai2-27b \
  --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec dflash2 --draft-tokens 5

A build listens on 127.0.0.1:8080 unless --host and --port say otherwise. Building shows how to make one.

Supported GPUs

NVIDIA GPUs from Volta (compute capability 7.0) to consumer Blackwell (12.0), from 8 GB of memory: Ternary Bonsai 2 27B runs on an 8 GB RTX 3060 Ti, and Qwen3.8-Flash-Next streams its experts from host memory or disk through 8 GB. The two Docker images each contain all NInfer targets for their CUDA version in a shared runtime: CUDA 12.9 includes Volta, and CUDA 13.4 includes Blackwell. The native release pipeline also assembles a portable Windows or Linux package with both CUDA backends: ninfer-<os>-x64-cuda12.9-13.4-<commit12>. Its small command executables share one runtime per CUDA family. The launcher selects CUDA 12 for Volta, CUDA 13 for consumer Blackwell, and prefers CUDA 13 on overlapping targets when the driver supports it. All selected --devices must fit one backend. Use --cuda-backend auto|12|13 to select it explicitly. No CUDA Toolkit installation is needed. Keep backends/ and help/ beside the executables. Linux also needs the recorded Debian system libraries. --help and --version do not load CUDA.

Every card by name, with what has run on it: Supported GPUs.

Click a section to expand it.

Architectures and builds — six architectures, two CUDA variants

Architecture	Compute capability	Build	CUDA	Docker tags	Status
Blackwell: GeForce RTX 50, RTX PRO Blackwell	12.0	120a	13.1 or newer	cuda13.4	measured; built-in device profiles
Ada Lovelace	8.9	89, or the 86 build	12.8 or newer	both, native 89	measured; built-in device profiles
Ampere	8.6	86 (the default)	12.8 or newer	both	measured (the reference card); built-in device profiles
Ampere, A100 class	8.0	80	12.8 or newer	both	built-in device profiles; not benchmarked
Turing	7.5	75	12.8 or newer	both	measured; built-in device profiles
Volta	7.0	70	12.8 or 12.9	cuda12.9	measured; built-in device profiles
Hopper; data-center Blackwell; GB10; Pascal and older		none			no build

What fits in your GPU memory — measured, from 8 GB to 48 GB

GPU memory	Runs	Measured on
8 GB	Ternary Bonsai 2 27B (24,576 tokens with rk4v4, no MTP); Qwen3.8-Flash-Next with its experts in host memory or on disk	RTX 3060 Ti
11 GB	Ternary Bonsai 2 27B (200,704 tokens with rk4v4; 143,360 with MTP)	RTX 2080 Ti
12 GB	Ternary Bonsai 2 27B (266,240 tokens); Qwen3.8-27B GSQ-RCO IQ3_S (8,192 tokens, no MTP)	RTX 3060
16 GB	Gemma 4 26B-A4B Q4_0 (61,440 tokens); Qwen3.8-27B IQ3_S (241,664 tokens); the official Qwen3.8-27B with --embedding-q4 --lm-head-q4	Tesla V100-SXM2-16GB
24 GB	the official Qwen3.8-27B and its MXFP8 tune; Qwen3.6-35B-A3B groupwise-int on an RTX 3090; Flash-Next with every expert on the GPU over two cards; DeepSeek-V4.1-Flash with its experts in host memory or on disk	RTX 3090, RTX 4090, TITAN RTX
32 GB	Qwen3.6-35B-A3B NVFP4 (Blackwell only)	RTX 5090
48 GB	Flash-Next Q2_0 with every expert on one card	L40S, RTX PRO 6000

Device profiles and limits — how a card gets its kernel routes

• Device profiles. A profile picks the kernel route of every operation and width. The binary carries profiles measured on 48 GPU models, from the Tesla V100 and RTX 2060 Super to the RTX PRO 6000 (the Profile column of supported GPUs); a built-in profile applies to the same GPU name, compute capability and SM count. Any other card is calibrated once at first start, which takes about half a minute to three and a half minutes, and the result is stored for later starts; ninfer-calibrate measures again after a driver or clock change (device profiles). The 70 build runs its own routes, so the V100 profiles hold only the GDN prefill switch and, for the 32 GB card, the Q8 MoE prefill frontier.
• Architecture limits. NVFP4 expert weights need a 120a build. The Turing and Volta builds replace instructions their GPUs lack and reshape kernels to their shared memory (Turing, Tesla V100). Configure accepts no other target, so Pascal and older, Hopper and data-center Blackwell have no build.
• Speed per card. Performance lists the measured rates of every card.

Artifacts

Each name links to its Hugging Face card; hf download REPO FILE.ninfer --local-dir models fetches the file. The Bonsai, GSQ-RCO, Flash-Next, Gemma 4, Flint and DeepSeek-V4.1-Flash artifacts use formats that only this engine reads. ✓ means the artifact carries the component, — that it does not.

Model	Size, GiB	Weights	Vision	MTP	DFlash2	Proposal head	Notes
Ternary Bonsai 2 27B	8.87	ternary, 2.125 bits	✓	✓	✓	✓	Bonsai-trained MTP head and DFlash2 adapter
Qwen3.8-27B	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the official artifact; the reference tables use it
Qwen3.8-27B GSQ-RCO IQ3_S	13.99	GGUF blocks, 3.5 bits	✓	✓	✓	✓	ISTA-DASLab's GGUF blocks byte for byte, Q6_K MTP head
Qwen3.8-27B, abliterated	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the official recipe
Qwen3.8-27B Uncensored, HauhauCS Aggressive	19.03	groupwise-int (Q4/Q5)	✓	✓	✓	✓	the tune's MTP head
Qwen3.8-27B MXFP8-CRACK	16.96	groupwise-int	✓	✓	—	✓
Qwen3.6-35B-A3B NVFP4	20.39	NVFP4 experts, Q8 projections	✓	✓	—	✓	RedHatAI's NVFP4 experts code for code; sm_120a GPUs only
Qwen3.8-Flash-Next GSQ-RCO Q2_0	38.49	GGUF blocks, 2.4 bits	✓	✓	—	—	shared-Q8_0 MTP; reads the n-gram table
Qwen3.8-Flash-Next GSQ-RCO IQ3_S	54.50	GGUF blocks, 3.5 bits	✓	✓	—	—	shared-Q8_0 MTP; reads the n-gram table
Qwen3.8-Flash-Next Coder GSQ-RCO IQ1_M	31.02	GGUF blocks	✓	✓	—	—	expert-pruned coding build, 256 experts per layer; shared-Q8_0 MTP; reads the n-gram table
Gemma 4 26B-A4B it QAT	14.75	GGUF Q4_0 blocks	✓	✓	—	—	Google's QAT GGUF byte for byte; the MTP column is Google's QAT assistant
Flint UD-Q4_K_XL	23.38	GGUF blocks, Q8_0 to IQ3_S	—	—	—	—	Unsloth's UD-Q4_K_XL GGUF byte for byte; drafts by context lookup
DeepSeek-V4.1-Flash-1017 GSQ-RCO IQ1_M	117.18	GGUF blocks, IQ1_S to IQ2_S experts	✓	—	—	—	ISTA-DASLab's GGUF byte for byte; experts in host memory or on disk; reads the Engram table; the release's vision tower (images); the release's DSpark drafter in beamster's STQ encoding (--spec dspark), or context lookup

The three Flash-Next releases read one shared n-gram table, which the model names by digest: the n-gram table (26.92 GiB, unchanged IQ4_NL rows plus an optional broad hot-row profile), passed with --ngram-table. DeepSeek-V4.1-Flash reads its own Engram table (103.00 GiB, the release's IQ4_NL rows) the same way.

The official NInfer artifacts (Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-27B, groupwise-int and nvfp4) load too; the documentation map links them. Weight conversion shows how the Bonsai and GSQ-RCO artifacts are built.

Highlights

Ternary Bonsai 2 27B decodes 397 tok/s on an RTX 5090 with DFlash2 and holds a 978,944-token context with every planted code found. Qwen3.8-Flash-Next decodes 138 tok/s with every expert on one RTX PRO 6000, and 96 tok/s with its experts in host memory on one RTX 5090. DeepSeek-V4.1-Flash decodes 52 tok/s on one RTX 5090, 48 on one RTX PRO 6000 and 39 on one RTX 4090 with its 106.7 GiB of experts in host memory. Every table of every card is on the performance page.

Click a section to expand it.

Ternary Bonsai 2 27B — up to 397 tok/s, a million tokens of context

Measured in September 2026, one card each, greedy, one request unless the column says otherwise.

Card	Short chat, DFlash2 7 drafts (tok/s)	After a 261K-token document (tok/s)	261K-token prompt, to the first token	Largest context, all three codes found	Eight requests at once, MTP 3 (tok/s)
RTX 3090	202	90	215 s	970,752	551
RTX 4090	256	123	102 s	958,464	824
RTX 5090	397	218	82 s	978,944	1,063
RTX PRO 6000	381	218	78 s	1,048,576, two of three codes	1,155

The RTX PRO 6000 (96 GB) starts at the engine's ceiling of 1,048,576 tokens with every KV format and drafter. Filled to it, the model misses the code at 90% (about 943K); up to about 880K it found every code on every card.

Qwen3.8-27B — up to 237 tok/s, 872,448 tokens of context filled

The official groupwise-int artifact, measured in September 2026, one card each, greedy, one request unless the column says otherwise.

Card	Short chat, DFlash2 7 drafts (tok/s)	Largest context, all three codes found	Eight requests at once, MTP 3 (tok/s)
RTX 3090	118	417,792	329
RTX 4090	149	405,504	442
RTX 5090	236	872,448	690
RTX PRO 6000	237	1,048,576, two of three codes	739

• RTX PRO 6000 against RTX 5090. Re-measured in October beside an RTX 5090, both at 600 W, the PRO 6000 came back within 2% of September wherever the drafts accepted the same share. One request's decode is bound by memory bandwidth, and both cards have 1.79 TB/s of GDDR7 (Bonsai 2 without speculation: 167.7 against 170.4 tok/s). The PRO 6000's 188 SMs against 170 show where compute decides: the 261K prompt is 2% faster and eight requests at once 6 to 7% faster (re-check).
• Draft length. DFlash2 with seven drafts is fastest on short answers; after long documents the best count is three to seven. MTP runs up to fifteen drafts and is fastest at three to five.

Qwen3.8-Flash-Next — 138 tok/s with the experts on a GPU, 98 tok/s from host memory

The Q2_0 release (35.9 GiB model and 26.8 GiB n-gram table), one request, greedy, October 2026. Decode of a short answer, in tok/s:

Card	Every expert on the GPUs	Experts in host memory	Experts on disk, in the page cache	Experts on disk, cold
RTX PRO 6000	138 (one card)	55	76	35
RTX 5090	136 (two cards)	98	72	38
RTX 4090	97 (two cards)	74	47	21
RTX 3090	90 (two RTX 3090 Ti)	39	28	20

Prefill of a 4,463-token prompt, in tok/s:

Card	Every expert on the GPUs	Experts in host memory	Experts on disk, in the page cache	Experts on disk, cold
RTX PRO 6000	2,939	1,297	2,024	1,000
RTX 5090	3,809	3,290	1,827	1,108
RTX 4090	3,608	1,801	1,144	558
RTX 3090	1,504	663	503	474

• The host matters. Host and disk rows depend on the host as much as on the card. The RTX 5090 host had PCIe 5.0 x16, the RTX PRO 6000 and RTX 4090 hosts PCIe 4.0 x16, and the RTX 3090 host PCIe 3.0 x16 and two sockets (the test ran pinned to the GPU's node). The host, disk and cold columns of the three GeForce cards were measured on October 26, the rest on October 5 and 6.
• Prompt spans. With host or disk experts a prompt prefills in spans of up to eight chunks, so each layer's missing experts cross the bus once per span. From the CLI with Q2_0 disk experts in the page cache, a 7,680-token prompt prefilled at 1,830 tok/s on the RTX 5090 and the answer decoded at 115 tok/s, against 450 and 32 chunk by chunk (three cards).
• Other releases and cards. IQ3_S, the Coder IQ1_M build and the 8 and 12 GB cards are in the expert placements. Six reasoning requests at once on two RTX 5090s produced 193 tok/s of output with Q2_0.

DeepSeek-V4.1-Flash — 52 tok/s on one RTX 5090, 48 on one RTX PRO 6000 (DSpark: 1.3 to 2.1 times as fast), 34 to 39 on one RTX 4090

ISTA-DASLab's GSQ-RCO IQ1_M GGUF (112.7 GiB, and its 103.0 GiB Engram table), one request, greedy, thinking off, one GPU, October 2026. The routed experts stay in pinned host memory or in the file, the card caches as many as fit, and CPU threads compute a share of each decode call's missing host experts:

Card	Experts cached on the card	Host experts: short answer (tok/s)	after 8,086 tokens (tok/s)	prefill of 8,086 tokens (tok/s)	Disk experts, page cache warm: short answer (tok/s)
RTX PRO 6000	74%	48.0	45.3	1,310	37.4
RTX 5090	15%	51.5	42.2	912	16.7
RTX 4090	7%	33.8	27.5	915	9.5
RTX 3090	7%	19.8	15.2	416	6.0

The RTX PRO 6000 row is an earlier build that ran prompts chunk by chunk and copied every missing expert; the others ran on October 26 on hosts with PCIe 5.0 (RTX 5090), 4.0 (RTX 4090) and 3.0 (RTX 3090) links (per card). On an RTX 4090 host with 64 CPU cores the build decodes at 38.9 and 31.4 tok/s, against 33.5 and 26.6 when it copies every miss (the CPU share); there Strata 0.2.6 decodes the same GGUF at 34 to 40 tok/s, computing about half the experts on its CPU threads, and prefills at 635 to 690 tok/s (measurements). With host experts the server peaks at 116 GiB of host memory. On October 27, with the release's DSpark drafter (--spec dspark, 3.58 GiB on the card in the STQ encoding the artifact carries), an RTX PRO 6000 decoded five requests 1.28 to 2.07 times as fast as without it, and an RTX 4090 decoded code and reasoning 1.19 and 1.21 times as fast, the other three requests 0.85 to 1.02 times, prefilling at plain speed (the drafter). With --vision it reads images through the release's vision tower (images).

Features

Click a section to expand it.

Models and weight formats — Qwen3.5 family, GGUF block formats, Ternary Bonsai 2, Flash-Next, Gemma 4, Flint, DeepSeek-V4.1-Flash

• Qwen3.5 family. Dense and MoE (Qwen3_5ForCausalLM, Qwen3_5MoeForCausalLM) with MTP, DFlash (35B-A3B) and DFlash2 (Qwen3.8-27B) drafters, Vision for images and video, and groupwise-int, FP8, NVFP4 and ternary weights. Native MoE routed experts can stay in pinned host memory or the artifact's files, brought into whole-layer GPU banks with an optional device cache (expert residency). NVFP4 expert banks prefill through W4A4 and need a 120a build.
• GGUF block formats. Releases that pick a ggml type per tensor, such as ISTA-DASLab's GSQ-RCO, convert without requantization and multiply their blocks in place. The 3.5-bit Qwen3.8-27B IQ3_S release holds 10.95 GiB of weights instead of 15.9, scores WikiText-2 perplexity 7.071 (the official artifact: 7.286) and decodes at 59.9 tok/s against the official artifact's 40.3 on an RTX 3090 (GGUF block formats).
• Ternary Bonsai 2 27B. PrismML's ternary Qwen3.8-27B runs from t2_g128_fp16 weights: 2.125 bits per weight, imported without rounding, with the Hadamard rotations fused into the norms and gates and integer activations for its projections. The bonsai2_27b_ternary recipe adds ProCreations' MTP head and DFlash2 adapter and an exact proposal head.
• Qwen3.8-Flash-Next. The 125B-parameter MoE (512 experts, about 6B active) from the GSQ-RCO GGUF releases, with its experts on the GPUs, in pinned host memory with a GPU cache (--expert-residency host) or in the artifact's files read into a GPU cache (--expert-residency disk, under 1 GB of RAM). It serves up to eight requests with prefix reuse, structured output, images and video, and MTP drafting. On two RTX 5090s the Q2_0 release scores 93.3% on AIME 2025 and 86.4% on GPQA-Diamond (Qwen3.8-Flash-Next).
• Gemma 4 26B-A4B. Google's QAT Q4_0 GGUF release, imported block for block (13.4 GiB), or a mixed-format GGUF of it: the sliding-window and full-attention blocks (the latter caching values only), the dense MLP beside 128 routed experts, every --kv-dtype format, up to eight concurrent requests with prefix reuse through host memory and shared system prompts, tool calls and structured output, images and video through the release's vision tower (on the GPU, streamed from pinned memory or on CPU threads), and Google's multi-token-prediction assistant for speculative decoding. On an RTX 3090 it prefills a 10,000-token prompt at 6,045 tok/s and decodes at 215 tok/s, 190 after 10,000 tokens, against llama.cpp's 4,697, 169 and 152 on the same GGUF; with two drafts a round chat answers decode at 291 tok/s instead of 209 (Gemma 4).
• Flint. Recursive's 39.8B coding MoE (2.5B active), from a llama.cpp GGUF such as Unsloth's UD-Q4_K_XL, imported block for block (23.4 GiB): sliding-window and position-free full-attention blocks with learned sinks, 256 routed experts beside a shared one, up to eight concurrent requests with prefix reuse through host memory, the model's own reasoning, GLM-4.7 tool calls, structured output, typed decisions, context-lookup drafting, and experts on the GPU, in pinned host memory or in the artifact's files. On an RTX 5090 it prefills a 10,000-token prompt at 14,656 tok/s and decodes at 348 tok/s, 315 after 10,000 tokens, against llama.cpp's 8,679, 223 and 209 on the same GGUF; on an RTX 3090, with its experts in host memory, at 5,755, 161 and 140 against llama.cpp's 2,553, 80 and 77 with four blocks' experts on the CPU (Flint).
• DeepSeek-V4.1-Flash. DeepSeek's 40-block MoE (384 routed experts, 6 per token) from ISTA-DASLab's GSQ-RCO IQ1_M GGUF, imported block for block, with its Engram table: four-stream hyper-connections, latent attention over a 128-position window and indexer-selected compressed latents with the release's FP8 and FP4 roundings, Engram rows read from the table's file or RAM, up to eight concurrent requests with prefix reuse through host memory, reasoning, DSML tool calls, structured output, typed decisions, images through the release's vision tower, the release's DSpark drafter and context-lookup drafting. Its 106.7 GiB of routed experts stay in pinned host memory or in the artifact's files, read into bounded buffers rather than mapped, and CPU threads compute a share of each decode call's missing host experts. Prompts run block by block in spans of up to 16,384 tokens, so each block's experts cross the bus once per span. On one RTX 4090 with its experts in host memory it decodes at 31.4 to 38.9 tok/s and prefills 8,086 tokens at 938 tok/s; on the same host Strata 0.2.6 decodes at 34 to 40 tok/s and prefills at 635 to 690 (DeepSeek-V4.1-Flash).
• CPU share of host and disk experts. With Flash-Next, Gemma 4, Flint or native Qwen3.5/3.6 MoE experts off the GPU, or DeepSeek-V4.1-Flash's experts in host memory, CPU threads compute a share of each decode call's cache misses from host memory while the GPU copies the rest, bit for bit equal to the GPU's products (--expert-dma-share auto). The pool takes min(physical cores, usable CPUs / 2) threads by default, where usable CPUs are the affinity mask bounded by a container's CPU quota (15 threads on a 30-CPU quota); --expert-cpu-threads N sets another count (common options).

Speculative decoding — MTP, DFlash, DFlash2, DSpark, n-gram copying and context lookup

• MTP, DFlash and DFlash2 with 1 to 15 drafts; the artifact's proposal head drafts by default when it stores one. Under MTP and DFlash a sampled request keeps a draft only while it equals the model's own keyed draw, so every emitted token is the model's sample; DFlash2 verifies its proposal distribution by rejection sampling.
• Gemma 4 drafts with Google's multi-token-prediction assistant (--spec mtp, 1 to 7 drafts), four blocks that read the model's own KV cache; its verification rows are bit for bit plain decode rows, so the output is the one plain decoding gives.
• DeepSeek-V4.1-Flash drafts with the release's DSpark drafter (--spec dspark, 1 to 5 drafts kept while its confidence holds): three blocks that read the backbone's last three blocks' streams. Its verification rows are bit for bit plain decode rows. In beamster's STQ encoding, which the published artifact carries (3.58 GiB on the card), it decodes 1.28 to 2.07 times as fast on an RTX PRO 6000; on an RTX 4090, whose expert cache it shrinks, it decodes code and reasoning 1.19 and 1.21 times as fast and other requests 0.85 to 1.02 times.
• Flint, which has no drafter, and DeepSeek-V4.1-Flash without --spec draft by context lookup (--lookup-ngram N, 1 to 7 drafts): what followed the last N tokens' earlier occurrence, verified in one pass of decode rows.
• --adaptive-mtp picks each round's width from measured draft survival and round cost.
• N-gram copy drafting beside the neural drafter (on with --spec) verifies up to 15 tokens copied from earlier prompt, tool-result or output text (ngram copy proposals).

Context and memory — nine KV formats, up to 1,048,576 tokens, a context cache with a disk tier

• Nine KV formats, from bf16 to rk2v4-e8 at 216 bytes per token and KV head; rk8v4 is the all-round choice and rk4v4 holds the native 262,144-token window of Qwen3.8-27B on a 24 GB card (context and memory).
• Up to 1,048,576 tokens of context, four times the native window, with plain RoPE or YaRN (--rope-yarn, for Qwen3.5, Qwen3.8-Flash-Next and Gemma 4; Flint's full-attention blocks carry no rotation and DeepSeek-V4.1's compressed blocks the release's own YaRN, so neither takes scaling).
• A hybrid prefix cache by default, which shares content-addressed KV blocks and sparse state snapshots across requests on the device and in pinned host memory; or a checkpoint catalog (--use-original-prefix-caching) with an optional disk tier that survives restarts (--disk-kv-path, --disk-kv-restore) and endpoint and branch anchors that let an edited or regenerated turn resume where it left the conversation.
• Precision trades that buy context: --embedding-q4, --lm-head-q6, --gdn-state-fp16 (quality trades).
• A token embedding kept in mapped host memory (--embedding-residency host), which frees its VRAM for context with no change to the output.

Serving — OpenAI, Anthropic and llama.cpp APIs, typed decisions, model suspend, several models behind one server

• OpenAI Responses and Chat Completions, Anthropic Messages, llama.cpp's /completion, /tokenize, /detokenize, /apply-template, /v1/rerank, /metrics, /slots and /props, streaming, tool calls, structured output through xgrammar (--structured-output), token log probabilities and prompt grafts (HTTP serving).
• Typed decisions. /v1/systemone (TypeSafe's System One API, as llama.cpp, vLLM and Ollama serve it) and /v1/decisions (OpenAI's Decisions API) answer choice, yes/no and score questions about a state with the served chat model: one prompt per question, the answer read from the first token's exact label log probabilities, the state prefilled once and every question run after it as a branch of one read; up to 200 questions, 255 options, Runoff's order-independent choices, a conversation as the state, images and video as data URLs (with --vision), and a margin by which a client escalates near ties (typed decisions).
• Model suspend. --model-suspend lets an idle server give its device memory back without exiting and take it again on the next request, retained conversations included. Ternary Bonsai 2 27B on an RTX 3090: 7.9 GiB down to 0.3 GiB in 0.33 s, back in 1.1 s (model suspend).
• Several models behind one server. With --models-dir or a llama.cpp-style --models-preset, ninfer-serve is a router with llama.cpp's model API; with --model-suspend a model sleeps instead of unloading: two 27B models on one 24 GB RTX 3090 swap in 1.8 s instead of an 11.6 s cold load (several models).
• Control vectors and direction LoRA adapters. Qwen3.5-core models load llama.cpp control vectors and the rank-one LoRA adapters that abliteration tools write, such as AtomicChat's for Ternary Bonsai 2 27B; a request sets adapter scales through llama.cpp's lora field (LoRA adapters and control vectors).
• GET /stats and a monitoring dashboard, request logs with rotation, and llama.cpp's WebUI, built into the Linux packages and the Docker images.

Kernels and devices — route profiles per card, pipeline stages, FP8 and NVFP4 tensor-core routes

• Device route profiles. Each card's measured profile picks the kernel route for every operation and width: built in for the cards in supported GPUs, measured once at first start on any other (device profiles).
• Pipeline stages. --devices 0,1,... splits the layers over up to eight GPUs on Linux, each stage owning its weights, KV and state; the cards may differ where the build runs on each, such as RTX 30 with RTX 40 cards on the 86 build (pipeline stages). --tensor-parallel divides every layer over the devices instead: each holds a share of the GDN heads, of the attention's KV heads with their cache and of the dense FFN rows or MoE experts in any weight format, sized to the card, and reads only those per token (tensor parallelism), Qwen3.8-Flash-Next its GDN key heads, attention KV heads and routed experts. Gemma 4, Flint and DeepSeek-V4.1-Flash divide only their routed experts this way.
• FP8 A8 and NVFP4 W4A4 tensor-core routes on every 120a build, an FP4 tensor-core prompt kernel for an NVFP4 KV cache, and programmatic dependent launches on Blackwell.

Docker

The release pipeline packages two Debian 13 / Linux x64 images from the native Linux archives. Each image contains one executable with all NInfer architectures supported by its CUDA variant. The image build does not compile NInfer again.

Image tag	Toolkit	Embedded GPU architectures
ghcr.io/iamwavecut/ninfer-all:cuda12.9	CUDA 12.9.2	sm_70, sm_75, sm_80, sm_86, sm_89
ghcr.io/iamwavecut/ninfer-all:cuda13.4	CUDA 13.4.3	sm_75, sm_80, sm_86, sm_89, sm_120a

Choose cuda12.9 for Volta and cuda13.4 for Blackwell. Both cover Turing, Ampere and Ada. Both support a layer pipeline (--devices) across different architectures included in that image. The host needs a compatible NVIDIA driver and the NVIDIA Container Toolkit. Windows archives, Linux archives and Docker images use the same two architecture sets:

Each archive and image contains all device images inside its outline. These are the project's build targets, not the complete list of architectures supported by NVIDIA's Toolkits. See the release process for build and verification status.

Commands, volumes and tags — what the container runs

• Tags. cuda12.9 and cuda13.4 update after both Linux CUDA builds and both Docker jobs succeed for the current master. Windows builds do not block them. Pin a build with cuda12.9-<commit12> or cuda13.4-<commit12>, where commit12 is the first 12 hexadecimal characters of its commit SHA. Both images must build before either floating tag moves. The pipeline no longer updates latest or multi tags.
• Volumes. /models holds artifacts, /cache the device profile measured on first start, and /grafts/<model>/ optional prompt grafts.
• Verification. The new image variants still need a complete build and GPU inference checks. Earlier image tests on RTX 3090 and RTX 5090 do not establish these variants' runtime behavior.

The container's command chooses what runs:

command	runs
serve, ninfer, perplexity, calibrate [args]	that binary with any artifact and flags; serve listens on 0.0.0.0:8080 unless given --host or --port
run <model> [profile] (the default: run qwen38-27b)	the launcher profiles of scripts/run.sh for the official qwen38-27b and qwen36-35b-a3b, with its NINFER_* overrides (-e NINFER_SPEC=mtp, -e NINFER_CONTEXT=131072, ...)
download <model>	scripts/download-model.sh into /models: qwen38-27b, qwen36-27b or qwen36-35b-a3b

The official Qwen3.8-27B with the measured tuned profile, on http://localhost:8080/v1:

docker run --rm -v "$PWD/models:/models" ghcr.io/iamwavecut/ninfer-all:cuda13.4 download qwen38-27b
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all:cuda13.4 run qwen38-27b

compose.yaml wires the GPU, the port and the volumes for the same: docker compose run --rm ninfer download qwen38-27b, then docker compose up -d. NINFER_CUDA_TAG=cuda12.9 docker compose up -d selects the Volta-capable image; Compose defaults to cuda13.4. The published images use their multiarch binary automatically. docker build -t ninfer . builds a separate Ubuntu image from source (--build-arg ARCHS=86 for one architecture, --build-arg 'ARCHS=75;86;120a' for the multi-architecture build).

Building

scripts/fetch-media-sources.sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --target ninfer-serve ninfer-calibrate

The first command downloads the FFmpeg and dav1d releases that a Linux build compiles in.

CMAKE_CUDA_ARCHITECTURES takes one build from supported GPUs, or a list of them for a pipeline across cards of different architectures (several architectures).

Toolchains and options — CUDA versions per target, guides

120a runs the mma.sync compatibility path, which the ternary route needs. CUDA 12.8 or newer builds 75, 80, 86 and 89, and 70 needs CUDA 12.8 or 12.9. A 120a build needs CUDA 13.1 or newer, since 12.8 and 12.9 miscompile sm_120a kernels and configure refuses them. The Linux and Windows guides cover dependencies, the opt-in build options and the launcher scripts; tests and benchmarks have their own guides.

License

Apache-2.0, as upstream. The Bonsai artifact's weights come from PrismML, ProCreations and Qwen, all Apache-2.0; its card lists the notices. The Qwen3.8-Flash-Next artifacts carry the Qwen Community License 1.0 of their model.