Start with the project README to build NInfer, download a published artifact, and run the CLI or HTTP server.
| Document | Purpose |
|---|---|
| CLI | text, chat-history, image/video input, output streams, sampling, MTP, and common runtime options |
| HTTP serving | OpenAI Responses/Chat Completions, Anthropic Messages, state, streaming, token counting, authentication, and tool calls |
| Performance | RTX 5090 single-request and concurrent-decode results, MTP/DFlash measurements, and reproduction commands |
| CLI examples | committed text, multimodal, thinking, long-decode, and long-context inputs |
The executable --help output is the exact source for command-line option spelling and defaults.
| Model | Weights | Download | Versioned model card source |
|---|---|---|---|
| Qwen3.6-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.6-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.8-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.8-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.6-35B-A3B | groupwise-int |
Hugging Face | model card |
The active references under maintainer/ record current architecture, model,
artifact, and maintenance contracts. These files are not additional user workflows or installed
API documentation.
Runtime and Op references:
- Small-scale concurrent inference architecture
- Paged KV context storage, ownership, and capacity model
- Op admission, contracts, ownership, qualification, and performance rules
- ReplaySSM GDN technical reference
- Linear benchmark contract and registered suites
Artifact and model references:
- NInfer artifact container
- Persistent tensor numeric formats
- Persistent storage layouts
- Qwen3.6-27B model semantics
- Qwen3.6-27B artifact contracts, including NVFP4
- Qwen3.8-27B artifact contracts, including the NVFP4 target
- Qwen3.6-35B-A3B model semantics
- Qwen3.6-35B-A3B artifact contracts
Pending implementation work:
- Softmax Attention organization and migration describes the single target state for an unfinished source and public-contract cutover; it is not the current implementation map.