Root Mean Square Layer Normalization

Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Optimized CUDA kernels for NVIDIA GB10 Blackwell (sm_121, DGX Spark). RMSNorm + GELU. First sm_121 kernel on HuggingFace Kernel Hub.

Fused Triton kernels for Transformer inference: RMSNorm+RoPE, Gated MLP, FP8 GEMM — CPU-testable references, autotuning, and benchmarking

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

A 36M-parameter goldfish language model with a 10-second memory + pixel-art PWA desk pet. Runs in your browser, fully offline. Adopt it at den-sec.github.io/glublm/desk-pet/

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

Simple character level Transformer

Generative models nano version for fun. No STOA here, nano first.

Optimized Fused RMSNorm implementation with CUDA. Features vectorized memory access (float4), warp-level reductions, and efficient backward pass for LLM training

A non-official implementation of Qwen 3.5, as there doesn’t seem to be a paper or any code available that I can find, so I decided to implement it just for fun.

Production-grade Triton kernel fusing residual add + RMSNorm + packed QKV projection into a single GPU launch for decoder-only transformer inference (Llama-3, Mistral, Qwen2). +2.4% tok/s, -1.5 GB VRAM on A10G.

A from-scratch PyTorch LLM implementing Sparse Mixture-of-Experts (MoE) with Top-2 gating. Integrates modern Llama-3 components (RMSNorm, SwiGLU, RoPE, GQA) and a custom-coded Byte-Level BPE tokenizer. Pre-trained on a curated corpus of existential & dark philosophical literature.

Build an LLM in PyTorch: BPE tokenizer, GPT-1/2 + LLaMA, end-to-end train/infer

A transformer language model built from scratch, from byte-level BPE tokenization through pretraining and QA fine-tuning.

CUDA kernels for LLM decode-stage inference, built as a PyTorch extension with correctness tests and latency benchmarks.

🚀 Build your own LLM easily with OpenLabLM, a lightweight, hackable codebase tailored for hobbyists using a single consumer GPU.