English | 简体中文 | Deutsch | Español | français | 日本語 | 한국어 | Português | Русский
The AI-native, cloud-native file system.
Full POSIX semantics on top of object storage, with a built-in multi-tier distributed cache.
Built in Rust for AI training, inference, and AI agent platforms.
Quick Start · Documentation · Benchmarks · Community
Object storage is cheap and durable, but AI workloads expect a fast local file system. Curvine sits between the two: your data stays in S3 (or any compatible store), and your applications see a POSIX file system backed by a memory, SSD, and HDD cache.
- Fast enough for GPUs. In an MLPerf Storage v2.0 comparison on the same AWS instance types, open-source Curvine beat Alluxio Enterprise AI's submitted results on accelerator utilization in 6 of 8 runs.
- Real POSIX compatibility. The FUSE mount passed 1,129 Linux Test Project (LTP) file system tests with zero failures, so tools like
git, Vite, andinotifywork without changes. - Built for agents at scale. A single cluster holds 5 billion files. On Kubernetes, a new volume is just a
mkdir, so you can run 10,000+ stateful agent pods on EKS without per-volume cloud API calls. - Proven with LLM inference. AWS published a reference architecture for tiered KV cache on Amazon SageMaker HyperPod with Curvine.
- No lock-in. Files map 1:1 to object paths in S3. Data persisted to your bucket stays readable as plain objects, even when Curvine is down.
Start a single-node demo cluster (one master, one worker) with one command:
curl -fsSL https://curvineio.github.io/install.sh | bashThe script prints the Web UI address and the commands for inspecting and removing the containers.
helm repo add curvineio https://curvineio.github.io/helm-charts
helm repo update
# Curvine cluster
helm upgrade --install curvine curvineio/curvine -n curvine --create-namespace
# CSI driver, once the cluster is reachable
helm upgrade --install curvine-csi curvineio/curvine-csi -n curvine-system --create-namespaceSee the Helm charts for configuration and the CSI driver guide for creating a StorageClass.
Follow the build instructions to produce a package in build/dist, then:
cd build/dist
export CURVINE_MASTER_HOSTNAME=localhost
./bin/curvine-master.sh start
./bin/curvine-worker.sh start
./bin/curvine-fuse.sh start # mounts at /curvine-fuse
bin/cv report # check cluster status
bin/cv fs mkdir /hello
ls /curvine-fuseTo stop, run the same scripts with stop in reverse order: curvine-fuse.sh, curvine-worker.sh, then curvine-master.sh.
Web UIs: master at http://<host>:9001, worker at http://<host>:9002.
The Quick Start guide covers each step in detail.
| Use case | What Curvine does |
|---|---|
| AI agent platform storage | Gives tens of thousands of stateful agent pods their own POSIX workspaces, provisioned in milliseconds, with no per-node volume limits. |
| LLM training | Caches datasets and checkpoints next to GPU nodes so training spends less time waiting on I/O. |
| Model distribution | Distributes model weights across regions through the cache instead of pulling from object storage on every node. |
| Multimodal data lakes | Serves images, video, and Lance datasets over POSIX without copying them to local disk. |
| OLAP acceleration | Caches hot data for compute-storage separated engines such as StarRocks. |
| Multi-cloud caching | Puts one cache layer in front of object storage from different cloud providers. |
See each use case in action
AI agent platform storage
LLM training
Model distribution
Multimodal data lakes
OLAP acceleration
Multi-cloud caching
Curvine has four layers, color-coded in the diagram above:
- Access layer. The workloads on top: AI agent pods, training and inference jobs, OLAP and big-data engines, and the
cvCLI. - Protocol layer. Existing tools connect without code changes through POSIX (FUSE), an S3-compatible gateway, the Hadoop FileSystem API, Java/Python/Rust SDKs, or the Kubernetes CSI driver.
- Cluster core.
- The control plane is a Raft-replicated master that owns metadata, the namespace, scheduling, and cluster coordination. It also serves the Web UI and metrics.
- The data plane is a set of workers that serve data from a memory → SSD → HDD cache, promoting hot data and evicting cold data automatically.
- Storage layer. Object storage keeps the durable copy: AWS S3, Azure Blob Storage, Google Cloud Storage, Alibaba Cloud OSS, MinIO, any other S3-compatible store, or HDFS.
Metadata requests go to the master over RPC. Data flows directly between clients and workers. On a cache miss, workers read from object storage, and writes are persisted back to it. On Kubernetes, the CSI driver exposes the FUSE mount as a PVC, so creating a volume only creates a directory and takes milliseconds.
| Area | Details |
|---|---|
| Access | POSIX via FUSE (open, read, write, seek, rename, list), S3 gateway, Hadoop FileSystem API, Java/Python/Rust SDKs |
| Kubernetes | CSI driver with dynamic provisioning, Immediate binding, and volume expansion; Helm charts for the cluster |
| Storage backends | AWS S3, Azure Blob Storage, Google Cloud Storage, Alibaba Cloud OSS, MinIO and other S3-compatible stores, HDFS |
| Caching | Memory → SSD → HDD tiers, automatic hot-data promotion, eviction, and replication |
| Performance | Rust and Tokio async I/O, zero-copy data paths, no garbage collector; around 100 µs latency and over 100K QPS sustained |
| Scale | Up to 5 billion files in one cluster |
| Reliability | Raft-replicated metadata; files map 1:1 to object paths, so data stays readable directly from object storage |
| Observability | Built-in metrics and Web UI for every component |
40 concurrent clients. JuiceFS and OSS figures come from JuiceFS's published benchmark.
| Operation | Curvine (QPS) | JuiceFS (QPS) | OSS (QPS) |
|---|---|---|---|
| create | 19,985 | 16,000 | 2,000 |
| open | 60,376 | 50,000 | 3,900 |
| rename | 43,009 | 21,000 | 200 |
| delete | 39,013 | 41,000 | 1,900 |
256 KB reads on identical hardware. Alluxio figures come from Alluxio's own published comparison. "N/A" means Alluxio did not publish a result for that thread count.
| Threads | Sequential: Curvine (GiB/s) | Sequential: Alluxio OSS (GiB/s) | Random: Curvine (GiB/s) | Random: Alluxio OSS (GiB/s) |
|---|---|---|---|---|
| 1 | 2.2 | 0.6 | 0.3 | 0.0 |
| 2 | 3.7 | 1.1 | 0.7 | 0.1 |
| 4 | 6.8 | 2.3 | 1.4 | 0.1 |
| 8 | 8.9 | 4.5 | 2.8 | 0.2 |
| 16 | 9.2 | 7.9 | 5.2 | 0.4 |
| 32 | 9.5 | 8.8 | 7.8 | 0.3 |
| 64 | 9.2 | N/A | 8.7 | N/A |
| 128 | 9.2 | N/A | 9.0 | N/A |
In production big-data shuffle workloads, Curvine used over 90% less memory and over 50% less CPU than Alluxio.
More results, including FIO and small-file reads, are in the benchmark docs.
- Documentation
- Architecture deep dives
- Best practices
- DeepWiki: ask questions about the codebase
- Blog
- Chat: Slack or Discord
- Questions and ideas: GitHub Discussions
- Bugs and feature requests: GitHub Issues
- Using Curvine? Add your company to ADOPTERS.md.
- Project direction: see the roadmap and governance.
Contributions are welcome. Read the contributing guide to get started, and look for issues labeled good first issue.
Made with contrib.rocks.
"Curvine" comes from curvature engine, the faster-than-light drive in Liu Cixin's science fiction novel The Three-Body Problem. We picked it because the project is about making data access as fast as possible.
Curvine is a Cloud Native Computing Foundation sandbox project.
Curvine is licensed under the Apache License 2.0.