Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@
},
"metadata": {
"description": "Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.",
"version": "0.1.0"
"version": "0.1.1"
},
"plugins": [
{
"name": "inferencekit",
"source": "./plugin",
"description": "Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.",
"version": "0.1.0",
"version": "0.1.1",
"author": {
"name": "Stellon Labs"
},
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ These rules apply to any agent doing inference performance work with this reposi

## Tools

- CLI `inferencekit <command>`; MCP server `inferencekit mcp serve` (stdio; every command is also a tool with the same arguments; launcher `uvx --from 'inferencekit-cli[mcp]==0.1.0' inferencekit mcp serve`). Commands and flags: `inferencekit --help` and `inferencekit <command> --help`.
- CLI `inferencekit <command>`; MCP server `inferencekit mcp serve` (stdio; every command is also a tool with the same arguments; launcher `uvx --from 'inferencekit-cli[mcp]==0.1.1' inferencekit mcp serve`). Commands and flags: `inferencekit --help` and `inferencekit <command> --help`.
- Measure: `doctor`, `ceiling bw`, `roofline`, `bench rounds` (runs `bench serve` per engine per round, fresh process, balanced order: published numbers), `bench serve` (one client-timed call), `bench decode` (engine-timed; engines without a server), `bench image`, `bench audio` (client-timed media screening; contracts in `docs/media-benchmarks.md`), `compare` (matching benchmark kinds; load records are refused).
- Check: `gates run`, `prompts verify|show`, `quality trajectory|score|gsm8k|sampling`, `correctness`.
- Diagnose: `nsys capture|analyze`, `ncu capture|summarize`.
Expand Down
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
<div align="center">

<img src="assets/inferencekit-kitten.png" width="200" height="200" alt="KittenML kitten holding a GPU">
<img src="https://raw.githubusercontent.com/KittenML/inferencekit/main/assets/inferencekit-kitten.png" width="200" height="200" alt="KittenML kitten holding a GPU">

# inferencekit

**Turn your coding agent into an inference engineer. Make your models faster across any hardware.**

Benchmark, profile and optimize text, image, audio and video models with your coding agent.

[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/KittenML/inferencekit/blob/main/LICENSE)
[![PyPI: inferencekit-cli](https://img.shields.io/pypi/v/inferencekit-cli.svg?label=inferencekit-cli)](https://pypi.org/project/inferencekit-cli/)
[![Claude Code](https://img.shields.io/badge/Claude%20Code-plugin-d97757.svg)](#install)
[![Codex](https://img.shields.io/badge/Codex-plugin-111111.svg)](#install)
[![OpenCode](https://img.shields.io/badge/OpenCode-plugin-4a7bd0.svg)](#install)

[Install](#install) · [How it works](#how-it-works) · [Showcase](#results) · [Features](#features) · [Engines](https://github.com/kittenml/inferencekit-engines) · [Roadmap](#roadmap) · [Contributing](CONTRIBUTING.md)
[Install](#install) · [How it works](#how-it-works) · [Showcase](#results) · [Features](#features) · [Engines](https://github.com/kittenml/inferencekit-engines) · [Roadmap](#roadmap) · [Contributing](https://github.com/KittenML/inferencekit/blob/main/CONTRIBUTING.md)

</div>

Expand All @@ -29,7 +29,7 @@ inferencekit gives your agent the playbook and the tools of a senior inference e
3. **Find the bottleneck.** It selects the profiler for your hardware: Nsight on NVIDIA, ROCm tools on AMD, or Instruments and Metal on Apple. It measures the critical path and investigates memory, compute and launch costs, checking current vendor docs when support is uncertain.
4. **Investigate and implement, if requested.** Search current code, papers, compiler docs and engineering talks; inspect dispatch, IR and assembly when useful. Choose graph execution, fusion, matrix kernels, packing, state reuse or other techniques from measured bottlenecks. Independent implementations can run in parallel; one coordinator integrates changes, and measurements stay serialized per device. Separate authorized devices may run concurrently.
5. **Prove every step.** Every change gets a fast correctness check and a timed screen; milestones get the full proof: interleaved rounds against the tuned baseline, the anti-cheat gates and quality against a reference model. Changes that miss the agreed performance or quality requirements are reverted.
6. **Ship it.** You get the fastest proven configuration and a [`REPORT.md`](plugin/skills/reporting-perf-claims/references/wording-guide.md) with the result, quality checks, commands to install and run it, saved outputs, and details to reproduce the measurements.
6. **Ship it.** You get the fastest proven configuration and a [`REPORT.md`](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/reporting-perf-claims/references/wording-guide.md) with the result, quality checks, commands to install and run it, saved outputs, and details to reproduce the measurements.

## When it asks you

Expand Down Expand Up @@ -79,7 +79,7 @@ GPT-6.1 Terra used inferencekit to wire online FP8 into selected transformer pro

**Fastest decode in our tested vLLM, SGLang and llama.cpp panel.** Claude Opus 5.5 used inferencekit to build **Qwen3-8B on RTX 5090** with custom FP8 matrix-vector kernels, a CUDA graph for the decode step and fused attention. Single-user decode was measured with in-process timers and no speculative decoding; llama.cpp Q8_0 is a separate format comparison. [Design, results and limits](https://github.com/KittenML/inferencekit-engines/tree/main/qwen3-8b-fp8_rtx-5090/).

![Bar chart of speedups against engine baselines on the same hardware, including tuned vLLM and MLX-Audio: Gemma 4 26B-A4B on RTX 5090, 1.33× faster than tuned vLLM; LTX-2.5 on RTX PRO 6000, 1.25× faster than ComfyUI; Qwen3-8B on RTX 5090, 1.22× faster than vLLM with FlashInfer; Drama3 on Apple M4, 1.11× faster than tuned MLX-Audio; MOSS-Image-SFT on RX 9070 XT, 2.46× with quality secondary (FP8, 24 vs 50 steps; paired 95% CI 2.453–2.465×, lower bound 2.453×)](assets/hero-speedups.png)
![Bar chart of speedups against engine baselines on the same hardware, including tuned vLLM and MLX-Audio: Gemma 4 26B-A4B on RTX 5090, 1.33× faster than tuned vLLM; LTX-2.5 on RTX PRO 6000, 1.25× faster than ComfyUI; Qwen3-8B on RTX 5090, 1.22× faster than vLLM with FlashInfer; Drama3 on Apple M4, 1.11× faster than tuned MLX-Audio; MOSS-Image-SFT on RX 9070 XT, 2.46× with quality secondary (FP8, 24 vs 50 steps; paired 95% CI 2.453–2.465×, lower bound 2.453×)](https://raw.githubusercontent.com/KittenML/inferencekit/main/assets/hero-speedups.png)

<details>
<summary>Measurement details and agent time, tokens and cost</summary>
Expand Down Expand Up @@ -126,15 +126,15 @@ Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing

## Features

- **9 skills, 4 commands and 5 read-only subagents**, backed by [41 executable CLI/MCP operations](src/inferencekit/ops/).
- **Reusable run tools:** [ikstep](plugin/skills/optimizing-inference/references/remote-host.md) manages jobs on local or remote Linux hosts, readiness checks and concise status output.
- **9 skills, 4 commands and 5 read-only subagents**, backed by [41 executable CLI/MCP operations](https://github.com/KittenML/inferencekit/tree/main/src/inferencekit/ops/).
- **Reusable run tools:** [ikstep](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/optimizing-inference/references/remote-host.md) manages jobs on local or remote Linux hosts, readiness checks and concise status output.
- **Compact output:** CLI summaries and bounded MCP responses keep full evidence records on disk, with paging for large lists.
- **Speed-of-light ceilings:** measured memory bandwidth (`ceiling bw`) and exact bytes per token read from the checkpoint's safetensors headers (`roofline`), including NVFP4, FP8 and mixture-of-experts models.
- **Engine triage and tuning:** vLLM, SGLang, llama.cpp, TensorRT-LLM, ComfyUI, Diffusers and others. Only settings that the engine's own logs show in effect are counted.
- **Kernel work:** profiling routed by hardware, CUDA/HIP, Triton and Metal kernels, and torch.compile. The included [engine skeleton](plugin/skills/optimizing-kernels/skeleton/README.md) targets NVIDIA CUDA.
- **Kernel work:** profiling routed by hardware, CUDA/HIP, Triton and Metal kernels, and torch.compile. The included [engine skeleton](https://github.com/KittenML/inferencekit/blob/main/plugin/skills/optimizing-kernels/skeleton/README.md) targets NVIDIA CUDA.
- **One timer for every engine:** interleaved rounds in a balanced order, medians with 95% confidence intervals.
- **Anti-cheat gates:** caching, token counts, output length, rare tokens, sampling probes, prompt rendering, speculative-decoding exactness.
- **Quality checks:** reference quality and GSM8K for language models; latent and final-clip checks for video. Images and speech use client timing and structural checks plus an external semantic quality protocol; see [media benchmarks](docs/media-benchmarks.md).
- **Quality checks:** reference quality and GSM8K for language models; latent and final-clip checks for video. Images and speech use client timing and structural checks plus an external semantic quality protocol; see [media benchmarks](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md).
- **Budget control:** spending authorized up front, with hardware-hour, dollar and elapsed-time limits, or explicitly uncapped spending with a finite deadline. No repeated plan approval inside the agreed scope.
- **Reproducible output:** a ranked table labelled PASS, FAIL, UNPROVEN or NOT RUN, a launch command, a review page and the commands that reproduce every number.

Expand Down Expand Up @@ -179,7 +179,7 @@ python3 ~/.local/share/inferencekit/plugin/opencode/install.py
| Codex | `$inferencekit:optimizing-inference`, `$inferencekit:benchmarking`, `$inferencekit:profiling`, `$inferencekit:reporting-perf-claims` | `codex plugin list`; `codex mcp list` |
| OpenCode | `/optimize-inference`, `/bench`, `/profile`, `/perf-report`, and each skill as `/<skill-name>` | `opencode mcp list` |

Updating, removing, per-project installs and OpenCode's options: [docs/install.md](docs/install.md).
Updating, removing, per-project installs and OpenCode's options: [docs/install.md](https://github.com/KittenML/inferencekit/blob/main/docs/install.md).

## Try it

Expand All @@ -195,18 +195,18 @@ To tune existing engines without custom kernels:

In OpenCode the command is `/optimize-inference`. It asks only for missing preferences and spending authorization; a plan is optional. For a repeatable setup, start from an intake file: `inferencekit intake template --out intake.yaml`, then `/inferencekit:optimize-inference intake.yaml`. For unattended runs, set `non_interactive: true` and `budget.approved: true` with a cap.

More things to ask for: [docs/examples.md](docs/examples.md).
More things to ask for: [docs/examples.md](https://github.com/KittenML/inferencekit/blob/main/docs/examples.md).

## CLI and MCP server

```bash
uv tool install "inferencekit-cli==0.1.0"
uv tool install "inferencekit-cli==0.1.1"
inferencekit doctor # Hardware, runtime and profiler availability
inferencekit gates run --base-url http://127.0.0.1:8000 # anti-cheat gates on any OpenAI-compatible server
uvx --from 'inferencekit-cli[mcp]==0.1.0' inferencekit mcp serve # MCP server (stdio) for any MCP client
uvx --from 'inferencekit-cli[mcp]==0.1.1' inferencekit mcp serve # MCP server (stdio) for any MCP client
```

Every CLI command is also an MCP tool with the same arguments. Text benchmarks, gates and quality checks talk HTTP to OpenAI-compatible servers. Image and streamed speech measurements use the [media adapter contracts](docs/media-benchmarks.md).
Every CLI command is also an MCP tool with the same arguments. Text benchmarks, gates and quality checks talk HTTP to OpenAI-compatible servers. Image and streamed speech measurements use the [media adapter contracts](https://github.com/KittenML/inferencekit/blob/main/docs/media-benchmarks.md).

## Supported hardware and agents

Expand Down Expand Up @@ -237,8 +237,8 @@ Profiles and logs can contain environment variables, host names and paths: scan

## Contributing

We welcome bug reports, engine settings and new skills. See [CONTRIBUTING.md](CONTRIBUTING.md). Develop with `uv sync --all-extras`, then `uv run --all-extras pytest -q` (no GPU needed). Report security problems privately: [SECURITY.md](SECURITY.md).
We welcome bug reports, engine settings and new skills. See [CONTRIBUTING.md](https://github.com/KittenML/inferencekit/blob/main/CONTRIBUTING.md). Develop with `uv sync --all-extras`, then `uv run --all-extras pytest -q` (no GPU needed). Report security problems privately: [SECURITY.md](https://github.com/KittenML/inferencekit/blob/main/SECURITY.md).

## License

Apache-2.0: [LICENSE](LICENSE) and [NOTICE](NOTICE). Copyright 2026 Stellon Labs. inferencekit is an independent project, not affiliated with NVIDIA, any engine project or any model publisher. Product and model names belong to their owners. Every result holds for the hardware, driver, engine version, model and workload it was measured on; change any of them and the numbers can change. The software comes with no warranty, as the Apache-2.0 licence states, and you are responsible for the cost of hardware use with it.
Apache-2.0: [LICENSE](https://github.com/KittenML/inferencekit/blob/main/LICENSE) and [NOTICE](https://github.com/KittenML/inferencekit/blob/main/NOTICE). Copyright 2026 Stellon Labs. inferencekit is an independent project, not affiliated with NVIDIA, any engine project or any model publisher. Product and model names belong to their owners. Every result holds for the hardware, driver, engine version, model and workload it was measured on; change any of them and the numbers can change. The software comes with no warranty, as the Apache-2.0 licence states, and you are responsible for the cost of hardware use with it.
8 changes: 4 additions & 4 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ codex plugin add inferencekit@inferencekit
```

- **Skills:** `$inferencekit:optimizing-inference`, `$inferencekit:benchmarking`, `$inferencekit:profiling`, `$inferencekit:reporting-perf-claims`. Type `$` in Codex to select an inferencekit skill, then describe the task.
- **Check:** `codex plugin list` shows `inferencekit@inferencekit` as `installed, enabled`, and `codex mcp list` shows the server launched with `uvx --from inferencekit-cli[mcp]==0.1.0 inferencekit mcp serve`.
- **Check:** `codex plugin list` shows `inferencekit@inferencekit` as `installed, enabled`, and `codex mcp list` shows the server launched with `uvx --from inferencekit-cli[mcp]==0.1.1 inferencekit mcp serve`.
- **Update:** `codex plugin marketplace upgrade` refreshes the marketplace; if the new version is not picked up, remove the plugin and add it again. **Remove:** `codex plugin remove inferencekit@inferencekit`.
- Independent work can use Codex's native subagents; without delegation, the same tasks run sequentially. See [parallel execution and device isolation](../plugin/skills/optimizing-inference/references/parallel.md).

Expand All @@ -43,12 +43,12 @@ python3 ~/.local/share/inferencekit/plugin/opencode/install.py

## The CLI or the MCP server alone

- **CLI:** `uvx --from 'inferencekit-cli==0.1.0' inferencekit doctor`, or `uv tool install 'inferencekit-cli==0.1.0'`.
- **MCP server, for any MCP client:** `uvx --from 'inferencekit-cli[mcp]==0.1.0' inferencekit mcp serve` (stdio). Every CLI command is also a tool with the same arguments.
- **CLI:** `uvx --from 'inferencekit-cli==0.1.1' inferencekit doctor`, or `uv tool install 'inferencekit-cli==0.1.1'`.
- **MCP server, for any MCP client:** `uvx --from 'inferencekit-cli[mcp]==0.1.1' inferencekit mcp serve` (stdio). Every CLI command is also a tool with the same arguments.
- **Unreleased source changes:** the plugin's MCP configuration uses the published CLI version above, not the current checkout. New operations require a release or an MCP server installed from the matching source; check the advertised tool list before relying on them.
- **If the MCP server does not start** (no `uvx`, no network, a failed install): install the CLI pinned to the plugin's version and let the agent call it from the shell. The skills say how.
```bash
uv tool install "inferencekit-cli==0.1.0"
uv tool install "inferencekit-cli==0.1.1"
inferencekit doctor
```

Expand Down
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "inferencekit",
"version": "0.1.0",
"version": "0.1.1",
"description": "Make your model serve faster on your GPU, and prove it. Skills for Claude Code, Codex and OpenCode, plus a CLI and MCP server: your agent picks and tunes inference engines, can write GPU kernels, and backs every number with client-timed benchmarks, anti-cheat gates and quality checks.",
"author": {
"name": "Stellon Labs"
Expand Down
2 changes: 1 addition & 1 deletion plugin/.mcp.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"command": "uvx",
"args": [
"--from",
"inferencekit-cli[mcp]==0.1.0",
"inferencekit-cli[mcp]==0.1.1",
"inferencekit",
"mcp",
"serve"
Expand Down
2 changes: 1 addition & 1 deletion plugin/opencode/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ An `opencode.json` with comments is never edited; the installer prints these key
"mcp": {
"inferencekit": {
"type": "local",
"command": ["uvx", "--from", "inferencekit-cli[mcp]==0.1.0", "inferencekit", "mcp", "serve"],
"command": ["uvx", "--from", "inferencekit-cli[mcp]==0.1.1", "inferencekit", "mcp", "serve"],
"enabled": true,
"timeout": 120000
}
Expand Down
2 changes: 1 addition & 1 deletion plugin/skills/benchmarking/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Measure end-to-end inference latency, throughput, memory and cost a
license: Apache-2.0
compatibility: Needs the inferencekit CLI (Python 3.10+, uv or uvx) or its MCP server, each engine at a pinned version (any OpenAI-compatible server), and an NVIDIA GPU for the full tool set (elsewhere ceiling bw is NOT RUN; roofline with a numeric --gpu-bw needs no GPU). Written for Claude Code, Codex and OpenCode.
metadata:
version: "0.1.0"
version: "0.1.1"
---

# Benchmarking inference speed
Expand Down
2 changes: 1 addition & 1 deletion plugin/skills/evaluating-quantization/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Choose a compressed model format by measuring memory, complete-requ
license: Apache-2.0
compatibility: Needs inferencekit (Python 3.10+, uv or uvx), a compatible runtime and the workload's quality reference/adapter. Header-based fit research needs no GPU; measurements require authorized hardware. Text examples use text endpoints; media uses native pipelines.
metadata:
version: "0.1.0"
version: "0.1.1"
---

# Evaluating quantization
Expand Down
Loading