Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .gitmodules

This file was deleted.

8 changes: 6 additions & 2 deletions BENCHMARK_GB200.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@

## KDA (Kimi Delta Attention)

> The tables below are historical full-KDA measurements retained for reference.
> The current standalone intra-kernel microbenchmark has a different scope and
> configuration and does not reproduce these results.

### Fixed-Length (H=64, D=128, bf16)

| B | T | FLA Triton (ms) | cuLA (ms) | Speedup |
Expand Down Expand Up @@ -54,10 +58,10 @@ Summary (10 configs): **avg=1.34x**, min=1.21x, max=1.44x.
Summary (18 configs): **avg=1.39x**, min=1.37x, max=1.40x.


To reproduce:
Current intra-kernel microbenchmark (different scope):

```bash
python benchmarks/bench_kda.py --mode both
python benchmarks/bench_kda_chunk_intra.py
```

## Lightning Attention
Expand Down
8 changes: 6 additions & 2 deletions BENCHMARK_GB200_CUDA_130.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@

## KDA (Kimi Delta Attention)

> The tables below are historical full-KDA measurements retained for reference.
> The current standalone intra-kernel microbenchmark has a different scope and
> configuration and does not reproduce these results.

### Fixed-Length (H=64, D=128, bf16)

| B | T | FLA Triton (ms) | cuLA (ms) | Speedup |
Expand Down Expand Up @@ -54,10 +58,10 @@ Summary (10 configs): **avg=1.33x**, min=1.20x, max=1.39x.
Summary (18 configs): **avg=1.35x**, min=1.33x, max=1.37x.


To reproduce:
Current intra-kernel microbenchmark (different scope):

```bash
python benchmarks/bench_kda.py --mode both
python benchmarks/bench_kda_chunk_intra.py
```

## Lightning Attention
Expand Down
8 changes: 6 additions & 2 deletions BENCHMARK_GB300.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@

## KDA (Kimi Delta Attention)

> The tables below are historical full-KDA measurements retained for reference.
> The current standalone intra-kernel microbenchmark has a different scope and
> configuration and does not reproduce these results.

### Fixed-Length (H=64, D=128, bf16)

| B | T | FLA Triton (ms) | cuLA (ms) | Speedup |
Expand Down Expand Up @@ -54,10 +58,10 @@ Summary (10 configs): **avg=1.45x**, min=1.36x, max=1.57x.
Summary (18 configs): **avg=1.32x**, min=1.29x, max=1.35x.


To reproduce:
Current intra-kernel microbenchmark (different scope):

```bash
python benchmarks/bench_kda.py --mode both
python benchmarks/bench_kda_chunk_intra.py
```

## Lightning Attention
Expand Down
67 changes: 26 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

Linear attention mechanisms reformulate standard attention to use linear-time state updates instead of quadratic pairwise interactions, making them well suited for long-context LLM workloads. Recent variants such as [GLA](https://arxiv.org/abs/2312.06635), [KDA](http://arxiv.org/abs/2510.26692), [GDN](https://arxiv.org/abs/2412.06464), and [Lightning Attention](https://arxiv.org/abs/2405.17381) further improve expressiveness with gating, delta-style updates, and chunkwise decomposition.

**cuLA** provides hand-tuned CUDA implementations of these linear attention variants, targeting NVIDIA Blackwell (SM10X) and Hopper (SM90) GPUs. It is designed as a submodule of [flash-linear-attention (FLA)](https://github.com/fla-org/flash-linear-attention), sharing the same interface — adopting cuLA requires only a one-line import change. For ease of maintenance, cuLA is currently developed as a standalone library; the end goal is for users to seamlessly access these kernels through FLA. Since FLA already has a kernel dispatch mechanism in place, integration will be ready soon.
**cuLA** provides hand-tuned CuTeDSL implementations of these linear attention variants, targeting NVIDIA Blackwell (SM10X) and Hopper (SM90) GPUs. The runtime is a standalone kernel library: it does not import or vendor FLA, Triton, or a C++/CUDA extension. Complete operators and deliberately exposed atomic kernels are available through `cula.kernels`.

> **⚠️ Early Stage:** cuLA is in its early development phase. Many kernels still have significant room for optimization, and the API may evolve. We warmly welcome contributions from the community — whether it's performance tuning, new algorithm support, bug fixes, or architectural improvements. Every contribution helps push the boundaries of linear attention on modern GPUs!

Expand All @@ -32,17 +32,12 @@ Platform-independent wheels containing the CuTeDSL sources are available on [Git

Replace `<TAG>` with the release tag (e.g., `v0.2.0`) and `<VERSION>` with the version (e.g., `0.2.0`). Or download the `.whl` file directly from the [Releases page](https://github.com/inclusionAI/cuLA/releases) and install it with `pip install <filename>.whl`.

The cuLA wheel does not install FLA. Install `flash-linear-attention` before importing `chunk_kda`:

pip install flash-linear-attention

### Build from Source

**Clone cuLA & dependencies:**

```bash
git clone https://github.com/inclusionAI/cuLA.git
git submodule update --init --recursive
```

**Install PyTorch and CuTeDSL:**
Expand All @@ -64,9 +59,6 @@ pip install 'nvidia-cutlass-dsl[cu13]>=4.6.2,<4.9'
**Install cuLA & dependencies:**

```bash
# Install flash-linear-attention for benchmark repro
pip install -e third_party/flash-linear-attention

# Install cuLA
pip install -e . --no-build-isolation
```
Expand Down Expand Up @@ -108,28 +100,26 @@ See [the SM90 Lightning prefill pipeline](docs/lightning_attn_sm90.md) for the
warp-group roles, WGMMA dataflow, pipeline stages, recurrent-state placement,
and fixed/packed scheduling modes.

### KDA (Kimi Delta Attention) — Blackwell (SM10X)

Just change the import:
### KDA (Kimi Delta Attention) prefill

```python
import torch
from cula.kda import chunk_kda # <-- one-line change from fla.ops.kda
from cula.kernels.kda import kda_prefill

B, T, H, K, V = 2, 2048, 32, 128, 128
device = 'cuda'

q = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16, requires_grad=True)
k = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16, requires_grad=True)
v = torch.randn(B, T, H, V, device=device, dtype=torch.bfloat16, requires_grad=True)
q = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16)
k = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16)
v = torch.randn(B, T, H, V, device=device, dtype=torch.bfloat16)
g = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16) * 0.1 # gate (log space)
beta = torch.randn(B, T, H, device=device, dtype=torch.bfloat16).sigmoid()
A_log = torch.randn(H, device=device, dtype=torch.float32) * 0.01
dt_bias = torch.zeros(H * K, device=device, dtype=torch.float32)
init_state = torch.zeros(B, H, K, V, device=device, dtype=torch.float32)

# Forward
o, final_state = chunk_kda(
# Forward-only standalone operator
o, final_state = kda_prefill(
q=q, k=k, v=v, g=g, beta=beta,
A_log=A_log,
dt_bias=dt_bias,
Expand All @@ -141,28 +131,27 @@ o, final_state = chunk_kda(
lower_bound=-5.0,
)

# Backward
do = torch.randn_like(o)
o.backward(do)

print(f'Output shape: {o.shape}') # [2, 2048, 32, 128]
print(f'Final state shape: {final_state.shape}') # [2, 32, 128, 128]
```

**Notes:**
- `safe_gate=True` is required to leverage TensorCore acceleration.
- `kda_prefill` is forward-only. The modular SM100 training primitives are exposed separately rather than hidden behind an FLA autograd wrapper.
- `safe_gate=True` is required by this path.
- `beta` supports both `float32` and `bfloat16`; `initial_state` must be `float32`.
- `cu_seqlens` (for variable-length sequences) must be `int32`.

## Usage

See [USAGE.md](USAGE.md) for detailed usage examples and notes.
See [the standalone kernel inventory](docs/KERNEL_VALUE.md) for the public
kernel boundary and the overlap assessment against FlashInfer and cuDNN.

## Benchmarks

Benchmarks run on a single **NVIDIA GB200/H200** GPU with **PyTorch 2.9.1**, **Triton 3.5.1**.

FLA baseline: [flash-linear-attention v0.5.0](https://github.com/fla-org/flash-linear-attention/releases/tag/v0.5.0).
Historical result files use FLA as an external baseline. FLA is not a runtime or source-tree dependency.

**Blackwell (SM10X)**

Expand All @@ -179,11 +168,17 @@ See [BENCHMARK_H200.md](BENCHMARK_H200.md) for CuTe DSL FlashKDA results on an H
- **FlashKDA Prefill (Hopper):** **avg 2.72x** speedup over FLA across 28 fixed-length and variable-length configs, up to **7.56x**.
- **FlashKDA Intracard CP (Hopper):** **4.29x geo-mean** speedup over serial FlashKDA on 28 CP-engaged long-sequence configs, up to **7.83x**.

To reproduce the benchmark suites directly:
The commands below run the currently maintained benchmark entry points. The
Blackwell KDA command is an intra-kernel microbenchmark; it does not reproduce
the historical full-KDA headline numbers above, which used different shapes
and included the complete operator pipeline.

The comparison commands below require FLA to be installed separately; FLA is
not imported by the cuLA runtime package.

```bash
# Blackwell (SM10X)
python benchmarks/bench_kda.py --mode both
python benchmarks/bench_kda_chunk_intra.py
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Comment thread
icavan marked this conversation as resolved.
Resolved
python benchmarks/bench_lightning_attn_prefill.py --modes no_state varlen
python benchmarks/bench_la_decode_vs_fla.py --heads 64 --head-dim 128

Expand All @@ -205,11 +200,8 @@ python -m pytest -m cula_slow -q
# Default tests plus every extended CUDA case
python -m pytest -m cula_full -q

# Tests for modular KDA forward against FLA Triton implementation
python -m pytest tests/test_kda_sm100_chunk_vs_fla.py -v
# Tests for modular KDA forward against naive KDA reference
python -m pytest tests/test_kda_sm100_chunk_vs_naive.py -v
# Tests for the SM90 CuTeDSL two-kernel prefill + intracard CP (vs FLA)
# Optional reference tests (require a separately installed FLA)
# SM90 CuTeDSL two-kernel prefill + intracard CP
python -m pytest tests/test_kda_sm90_prefill_vs_fla.py tests/test_kda_sm90_intracard_cp.py -v
# Tests for Lightning Attention prefill on SM100
python tests/test_lightning_sm100_prefill.py
Expand All @@ -218,13 +210,6 @@ python -m pytest tests/test_lightning_attn_prefill_dispatch.py tests/test_lightn
# Tests for Lightning Attention decode
python -m pytest tests/test_lightning_decode.py -v

# The two chunk-vs-reference files also retain their narrower KDA fast/slow split.
# Fast (default) — representative correctness paths for default CI and local iteration
python -m pytest tests/test_kda_sm100_chunk_vs_naive.py tests/test_kda_sm100_chunk_vs_fla.py -v
# Slow — broader stress coverage for nightly or manual runs
python -m pytest -m kda_slow tests/test_kda_sm100_chunk_vs_naive.py tests/test_kda_sm100_chunk_vs_fla.py -v
# Full sweep (fast + slow) — run before submitting a PR
python -m pytest -m kda_full tests/test_kda_sm100_chunk_vs_naive.py tests/test_kda_sm100_chunk_vs_fla.py -v
```

<details>
Expand Down Expand Up @@ -252,7 +237,7 @@ See [REPO_LAYOUT.md](REPO_LAYOUT.md) for the full directory structure and a summ

## Roadmap

* [ ] Integrate into [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) via FLA's kernel dispatch mechanism
* [ ] Stabilize the standalone atomic-kernel contracts and publish per-kernel benchmarks.
* [ ] Polynomial approximation to mitigate the exponential bottleneck, as in [Flash-Attentiton-4](https://arxiv.org/abs/2603.05451).
* [ ] Larger chunk size and 2-CTA on SM10X for improved throughput.
* [ ] Continuous optimization via agentic methods such as [AVO](https://arxiv.org/abs/2603.24517).
Expand All @@ -262,7 +247,7 @@ See [REPO_LAYOUT.md](REPO_LAYOUT.md) for the full directory structure and a summ

**Train**

* [x] Modular KDA Forward (SM10X, compatible with [Kimi CP](https://github.com/fla-org/flash-linear-attention/blob/main/fla/ops/cp/README.md))
* [x] Modular KDA Forward (SM10X)
* [x] kda chunk intra
* [x] chunk gated delta h
* [x] recompute wu
Expand All @@ -284,7 +269,7 @@ See [REPO_LAYOUT.md](REPO_LAYOUT.md) for the full directory structure and a summ

* [ ] Fused KDA prefill kernel (SM10X)

* [ ] MTP support
* [x] KDA and Lightning Attention MTP decode/verify support

* [ ] More aggressive fusion of small neighboring kernels like cumsum for inference scenarios.

Expand Down
19 changes: 10 additions & 9 deletions REPO_LAYOUT.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,16 @@ cuLA/
│ ├── __init__.py
│ ├── utils.py # Architecture, stream-buffer, and cu_seqlens helpers
│ │
│ ├── kda/ # KDA public API, wrappers, autograd, and support kernels
│ │ ├── __init__.py # Lazy exports for chunk, prefill, and decode APIs
│ ├── kernels/ # Stable standalone lazy exports + static registry
│ │ ├── kda.py # KDA operators and atomic primitives
│ │ ├── lightning.py # Lightning operators
│ │ └── registry.py # Architecture/level/status metadata
│ │
│ ├── kda/ # KDA public wrappers
│ │ ├── __init__.py # Lazy exports for prefill and decode APIs
│ │ ├── prefill.py # Public FlashKDA prefill validation and routing
│ │ ├── flashkda.py # SM90 CuTeDSL K1+K2 wrapper, autograd shell, and CP selection
│ │ ├── chunk.py # SM100 modular chunk API + autograd
│ │ ├── chunk_fwd.py # SM100 forward orchestration
│ │ ├── chunk_intra.py # CuTeDSL forward intra + Triton backward intra
│ │ ├── chunk_bwd.py # Triton + FLA + CuTeDSL backward orchestration
│ │ └── wy_intra.py / wy_recompute.py # Triton WY preparation kernels
│ │ └── chunk_intra.py # Pure-CuTeDSL SM100 intra/WU composition
│ │
│ ├── lightning/ # [non-KDA] Lightning Attention public wrappers
│ │
Expand Down Expand Up @@ -47,7 +48,6 @@ cuLA/
├── tests/ # CPU registry tests and architecture-marked GPU tests
├── docs/ # Pipeline and backend-dispatch design notes
├── scripts/build_wheel.sh
├── third_party/flash-linear-attention/ # FLA baseline and reused operations
├── README.md USAGE.md REPO_LAYOUT.md RECOMMENDED_CODING_STYLE.md
└── setup.py pyproject.toml LICENSE
```
Expand All @@ -56,7 +56,8 @@ cuLA/

| Directory | Language | Description |
|-----------|----------|-------------|
| `cula/kda/` | Python / Triton | Public KDA APIs, wrappers, autograd, routing, and supporting Triton kernels. `kda_prefill` calls FlashKDA directly without a native fallback. See [`cula/kda/README.md`](cula/kda/README.md). |
| `cula/kernels/` | Python | Framework-independent public surface and static kernel inventory. Imports are lazy and do not compile kernels. |
| `cula/kda/` | Python / CuTeDSL | KDA prefill/decode wrappers and the pure-CuTeDSL SM100 modular composition. `kda_prefill` calls FlashKDA directly without a native fallback. See [`cula/kda/README.md`](cula/kda/README.md). |
| `cula/ops/kda/` | Python (CuTeDSL) | CuTeDSL KDA kernels organized into SM100 modular kernels, SM90 FlashKDA K1+K2 and intracard CP, decode, and experimental code. |
| `cula/ops/lightning/` · `cula/ops/experimental/` | Python (CuTeDSL) | `[non-KDA]` Lightning/linear-attention kernels and prototypes. Lightning prefill dispatches to the SM90 or SM100 backend from Q's device capability. |
| `cula/ops/{inv,ptx}.py` · `cula/ops/sm100/ptx.py` | Python | Shared low-level helpers used across operators. |
45 changes: 17 additions & 28 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ cuLA exposes several KDA entry points targeting different GPU architectures:

| Kernel | GPU | Import |
|---|---|---|
| Modular Forward | Blackwell (SM100) | `from cula.kda import chunk_kda` |
| Modular Intra/WU | Blackwell (SM100) | `from cula.kernels.kda import chunk_kda_fwd_intra` |
| Two-Kernel Prefill | Hopper (SM90) | `from cula.kda import flashkda_prefill` |
| Public Prefill | Hopper (SM90) / Blackwell (SM100) | `from cula.kda import kda_prefill` |

`chunk_kda` and `flashkda_prefill` each call a fixed implementation.
`chunk_kda_fwd_intra` and `flashkda_prefill` each call owned CuTeDSL implementations.
`kda_prefill` is the public FlashKDA wrapper and has no native-extension fallback.
The removed `kda_prefill_hopper`, `kda_prefill_hopper_opt`, and
`kda_prefill_hopper_auto` names should be migrated to one of these FlashKDA
Expand Down Expand Up @@ -61,50 +61,39 @@ with torch.inference_mode():
)
```

### Modular Forward (SM100 — Blackwell)
### Modular Intra/WU primitives (SM100 — Blackwell)

The modular forward kernel replaces sub-kernels of KDA in FLA (chunk_intra, chunk_delta_h, fwd_o, etc.) for easy integration with [Kimi CP](https://github.com/fla-org/flash-linear-attention/blob/main/fla/ops/cp/README.md).
This boundary accepts an already activated, chunk-cumulative FP32 gate and
returns the intra-chunk matrices plus W/U preparation tensors. Gate activation,
cumsum, recurrent-state propagation, and output projection remain explicit
caller-owned stages.

#### Example

```python
import torch
from cula.kda import chunk_kda
from cula.kernels.kda import chunk_kda_fwd_intra

B, T, H, K, V = 2, 2048, 32, 128, 128
device = 'cuda'

q = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16, requires_grad=True)
k = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16, requires_grad=True)
v = torch.randn(B, T, H, V, device=device, dtype=torch.bfloat16, requires_grad=True)
g = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16) * 0.1
q = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16)
k = torch.randn(B, T, H, K, device=device, dtype=torch.bfloat16)
v = torch.randn(B, T, H, V, device=device, dtype=torch.bfloat16)
gk = torch.randn(B, T, H, K, device=device, dtype=torch.float32) * 0.1
beta = torch.randn(B, T, H, device=device, dtype=torch.bfloat16).sigmoid()
A_log = torch.randn(H, device=device, dtype=torch.float32) * 0.01
dt_bias = torch.zeros(H * K, device=device, dtype=torch.float32)
init_state = torch.zeros(B, H, K, V, device=device, dtype=torch.float32)

o, final_state = chunk_kda(
q=q, k=k, v=v, g=g, beta=beta,
A_log=A_log, dt_bias=dt_bias,
initial_state=init_state,
output_final_state=True,
use_qk_l2norm_in_kernel=True,
use_gate_in_kernel=True,
w, u, qg, kg, Aqk, Akk = chunk_kda_fwd_intra(
q=q, k=k, v=v, gk=gk, beta=beta,
safe_gate=True,
lower_bound=-5.0,
)

# Backward is supported
o.backward(torch.randn_like(o))

print(f'Output shape: {o.shape}') # [2, 2048, 32, 128]
print(f'Final state shape: {final_state.shape}') # [2, 32, 128, 128]
print(w.shape, u.shape, Aqk.shape, Akk.shape)
```

**Notes**

- The backward pass is currently supported via FLA's implementation; further optimizations are on the roadmap.
- Compatible with [Kimi CP](https://github.com/fla-org/flash-linear-attention/blob/main/fla/ops/cp/README.md) via the `cp_context` parameter, same as in FLA.
- This is not a full KDA operator and has no autograd wrapper.
- Import individual primitives from `cula.kernels.kda` when integrating a custom pipeline.

---

Expand Down
Loading
Loading