⚡ SpecInferKit

Production-grade speculative decoding for large language models — accelerate LLM inference by 2–3× without quality degradation using Eagle draft models, Medusa multi-token heads, and FP8 quantization.

Features • Quick Start • Architecture • Benchmarks • Modules • Contributing

---

⭐ Love SpecInferKit? Star it on GitHub — it helps others discover the project!

---

Features

Capability	Description	Why It Matters
Eagle Drafting	Autoregressive draft model with tree-based speculation	2.6× speedup on LLaMA-3.1-405B
Medusa Heads	Multi-token prediction via parallel feedforward heads	Zero extra memory at inference
FP8 Quantization	8-bit float inference with < 1% accuracy loss	2× memory reduction
Online Training	Continuous fine-tuning of draft models during serving	Adapts to real-time traffic patterns
Distributed Serving	Multi-GPU speculative pipeline with load balancing	Linear scaling with GPU count
Extensible API	Plugin interface for custom draft models and verifiers	Bring your own architecture

---

Quick Start

# Install
pip install specinferkit

# Run with Eagle speculation (LLaMA-3.1-70B + draft model)
specinferkit serve \
  --model meta-llama/Llama-3.1-70B \
  --draft eagle \
  --draft-model draft-small

# Run with Medusa heads
specinferkit serve \
  --model meta-llama/Llama-3.1-70B \
  --draft medusa \
  --medusa-heads 4

from specinferkit import SpeculativeEngine

engine = SpeculativeEngine(
    target_model="meta-llama/Llama-3.1-8B",
    draft_model="draft-tiny",
    strategy="eagle",
)

output = engine.generate(
    "Write a blog post about AI speculation:",
    max_tokens=1024,
    temperature=0.7,
)

---

Architecture

flowchart TB
    subgraph Input["Input Processing"]
        A[Prompt Tokens] --> B[Token Embedding]
    end

    subgraph Draft["Draft Stage"]
        B --> C{Eagle / Medusa?}
        C -->|Eagle| D[Autoregressive Draft Model]
        C -->|Medusa| E[Multi-Head Predictor]
        D --> F[Draft Token Tree]
        E --> F
    end

    subgraph Verify["Verification Stage"]
        F --> G[Target Model Forward Pass]
        G --> H[Rejection Sampling]
        H --> I{Accepted?}
        I -->|Yes| J[Append All Tokens]
        I -->|No| K[Append up to Accepted Position]
    end

    subgraph Output["Output"]
        J --> L[Decoded Text]
        K --> L
        L --> M[Next Iteration]
        M --> A
    end

Loading

---

Benchmarks

Model	Baseline (tokens/s)	SpecInferKit (tokens/s)	Speedup	Memory (GB)
LLaMA-3.1-8B	48.2	124.6	2.6×	16 → 9
LLaMA-3.1-70B	12.1	31.8	2.6×	140 → 72
LLaMA-3.1-405B	2.8	7.3	2.6×	810 → 420
Mistral-7B	55.3	138.2	2.5×	14 → 8
DeepSeek-V3	8.9	23.5	2.6×	190 → 98

Benchmarks measured on NVIDIA H100 (80GB) with batch size 1, input length 512, output length 256.

---

Modules

specinferkit/
├── algorithms/          # Speculation strategies
│   ├── eagle.py         # Eagle draft model
│   └── medusa.py        # Medusa multi-head
├── trainer/             # Draft model training
│   ├── base.py          # Base trainer
│   ├── online.py        # Online fine-tuning
│   └── distributed.py   # Multi-GPU training
├── quantization/        # FP8 compression
│   └── fp8.py           # FP8 quantizer
├── serving/             # Inference server
│   ├── server.py        # gRPC serving
│   └── client.py        # Python client
├── data/                # Dataset loaders
├── eval/                # Benchmarks
├── cli/                 # Command-line interface
└── utils/               # Shared helpers

---

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

• Report bugs — Open an issue
• Suggest features — Start a discussion
• Submit PRs — Fork and open a pull request

---

License

MIT

---

🌐 Crynge Ecosystem

All repos are free and open-source. ⭐ Star what you use!

Category	Repos
LLM & AI	SpecInferKit · AetherAgents · PromptShield
Marketing	AdVerify · Attributor · InfluencerHub · EdgePersona · AdVantage · BrandMuse · CampaignForge
Simulation	CivSim · EvalScope
Operations	OpsFlow

_{Built by Crynge · ⭐ Star us on GitHub!}