This document provides essential context, architectural principles, codebase structure, development standards, and testing procedures for AI agents and human engineers working on the HyperCache codebase.
HyperCache is a host-based caching and acceleration engine designed for high-performance AI training and inference clusters (PyTorch, Safetensors, Hugging Face).
- Network Storage Bottlenecks: Loading multi-gigabyte or terabyte model checkpoints from S3 or network-attached NFS during training job initialization or auto-scaling causes severe GPU idle time and high egress costs.
- Linux Page Cache & Host CPU Overhead: Traditional file system reads pass data through the Linux kernel VFS, page cache, and host CPU memory before reaching GPU VRAM.
- Cluster-Wide Coordination Friction: In multi-node GPU clusters, coordinating model caching across dozens of nodes requires a centralized, secure control plane with parallel dispatch.
- Striped Raw Block Acceleration: Auto-discovers physical/virtual NVMe drives (
/dev/vdc,/dev/vdd,/dev/nvme*) and stripes model weights into 4 MiB chunks directly across raw devices. - Direct Block DMA / GPUDirect Storage (GDS): Streams data directly from raw NVMe blocks or
O_DIRECTfilesystem arenas into GPU VRAM using Direct Block DMA (O_DIRECT,kvikio.CuFile), achieving hardware line-rate read throughput (~7 GB/s per PCIe Gen 4 NVMe, ~14 GB/s per PCIe Gen 5 NVMe). - Dual-Tier Metadata Architecture: In-memory Moka W-TinyLFU cache for sub-millisecond route resolution, backed by an atomic NVMe persistence manifest for instant warm-boot recovery (<20 ms).
- Dual Authoritative Backends: Native streaming ingestion from both Amazon S3 / S3-compatible object stores and NFS / Shared POSIX file systems.
- Centralized Cluster Controller & Coordinator: Dedicated control plane (
./hypercache-admin) for cluster topology discovery, aggregate NVMe capacity tracking, and parallel fleet-wide pre-loading (preload,evict,purge).
When modifying or extending this codebase, you MUST adhere to the following rules:
- Strict Zero-Emoji Policy:
- Do NOT use any emojis in code, comments, docstrings, terminal outputs, CLI logs, unit tests, commit messages, or documentation.
- Zero Network Exposure by Default:
- The worker daemon (
./hypercache start) must open ZERO network TCP ports by default. - By default, communication happens exclusively over the local Unix Domain Socket (
/run/hypercache/hypercache.sockor fallback/tmp/hypercache/hypercache.sock). - Prometheus exporter (port 9110) and Node Agent Fleet API (port 9115) must remain OFF unless explicitly enabled via flags or configuration.
- The worker daemon (
- Zero-Config Local Admin Ergonomics:
- Running
./hypercache-adminon the Controller host (127.0.0.1/ loopback) must work immediately with zero flags, zero tokens, and zero IPs required. - Remote access over the network requires Bearer token authentication (
cluster_auth_token/HYPERCACHE_AUTH_TOKEN).
- Running
- Preserve Backward Compatibility:
- Maintain compatibility for existing Python bindings (
fsspec,torch.load), IPC protocols, and TOML configuration keys.
- Maintain compatibility for existing Python bindings (
- Full Test Verification:
- All Rust unit tests (
cargo test) and Python unit tests (pytest) must pass cleanly before completing any task.
- All Rust unit tests (
hypercache-poc/
├── AGENTS.md # This AI contributor and developer guide
├── CLAUDE.md # Claude Code entry point; imports this file
├── CHANGELOG.md # Release notes per version
├── README.md # Public project README and architectural summary
├── GETTING_STARTED.md # Operator quickstart and cluster setup walkthrough
├── ARCHITECTURE.md # In-depth systems design document
├── build.sh # Compiles native release binaries (./hypercache, ./hypercache-admin)
├── install.sh # Complete environment installer (Rust, Python .venv, PyTorch dependencies)
│
├── configs/
│ ├── hypercache-agent.toml # Worker Node Agent configuration (drives, S3/NFS, cluster registration)
│ ├── hypercache-controller.toml # Central Controller configuration (network binding, auth token)
│ └── grafana/ # Grafana provisioning & dashboard templates
│
├── control-plane/ # High-performance Rust control plane & CLI suite
│ ├── Cargo.toml # Dependencies: tokio, axum, moka, reqwest, tower, clap, serde, crc32fast
│ ├── Cargo.lock # Committed: binaries need reproducible (--locked) builds
│ ├── rustfmt.toml # max_width = 120; CI enforces cargo fmt --check
│ ├── src/
│ │ ├── lib.rs # Library crate root; both binaries build on it
│ │ ├── access.rs # client_gid sharing of the socket, arenas and RAM-tier files
│ │ ├── main.rs # Binary: ./hypercache (Worker Node Daemon & Local CLI)
│ │ ├── bin/
│ │ │ └── admin.rs # Binary: ./hypercache-admin (Cluster Administration Suite)
│ │ ├── config.rs # Strongly typed configuration structs and TOML loader
│ │ ├── protocol.rs # JSON-lines IPC wire protocol over Unix Domain Socket (SyncStatus, ChunkDescriptor)
│ │ ├── server.rs # UDS server (bounded requests), commit/sha256 verification, write-back retry & recovery
│ │ ├── auth.rs # Constant-time Bearer token validation shared by controller and agent
│ │ ├── telemetry.rs # Prometheus metrics collector and optional HTTP exporter
│ │ ├── agent_server.rs # Worker Node Agent HTTP server (port 9115), streaming P2P/replica, heartbeat loop
│ │ ├── fleet.rs # CLI client functions for fleet administration commands
│ │ ├── bench.rs # `hypercache bench` end-to-end benchmark (explicit synthetic opt-in)
│ │ ├── check.rs # `hypercache check` pre-flight host readiness checks
│ │ ├── discovery.rs # Raw block device discovery & safety exclusion engine
│ │ ├── metadata.rs # Moka W-TinyLFU cache, extent allocator + quarantine, generations, manifest writer
│ │ ├── ram.rs # Tier 1 POSIX shared memory (/dev/shm) chunk pool
│ │ ├── controller/
│ │ │ └── mod.rs # FleetRegistry, strict node targeting, network security, parallel dispatch
│ │ └── ingestion/
│ │ ├── mod.rs # Backend router, single-flight ingest, digest, write-back flush
│ │ ├── striper.rs # Chunk striping across devices with rollback, CRC32 and fdatasync
│ │ ├── s3.rs # SigV4 S3 client: parallel ranged GET, multipart upload, ListObjectsV2
│ │ ├── nfs.rs # POSIX / NFS reader with path confinement (nfs_allowed_paths)
│ │ └── synthetic.rs # Synthetic benchmark payloads (opt-in only)
│ └── tests/
│ └── s3_live.rs # #[ignore]d live S3 tests (run by scripts/verify_release.sh)
│
├── scripts/
│ ├── verify_release.sh # Release verification harness (static, rust, python, e2e, s3, linux, image)
│ ├── e2e/hypercache_e2e.py # End-to-end daemon / fleet / S3 scenarios against the real binaries
│ ├── e2e/bare_metal_access.sh # Bare-metal Linux (root): raw loop device, arena capacity, non-root clients
│ ├── e2e/container_host_access.sh # Containerized daemon used by a non-root "host" client
│ ├── e2e/host_probe.py # Client probe run by the container and Kubernetes scenarios
│ ├── e2e/k8s_smoke.sh # Bundled K8s manifest on a throwaway minikube cluster
│ └── setup_mock_devices.sh # Creates sparse mock block device files in /tmp
│
└── client-py/ # Zero-copy Python AI client & framework integration
├── pyproject.toml # Python package specification
├── hypercache/
│ ├── __init__.py # Package entry point, auto-registers fsspec hook
│ ├── client.py # Unix Domain Socket IPC client
│ ├── datapath.py # Direct Block DMA / GDS direct I/O reader
│ ├── fsspec_fs.py # AbstractFileSystem implementation for torch.load/save
│ ├── kv_store.py # Content-addressed KV-Cache Prefix Block Store
│ ├── pipeline.py # Double-buffered pipelined GPU tensor loader
│ └── integrations/
│ ├── safetensors_loader.py # Zero-copy Safetensors & vLLM weight loader
│ └── dcp.py # PyTorch Distributed Checkpoint (DCP) StorageReader/Writer
└── tests/
├── test_client.py # Unit tests for UDS protocol and client metadata parsing
├── test_fsspec.py # Unit tests for fsspec file operations and streaming writes
├── test_integrations_and_kv.py # Unit tests for Safetensors/vLLM, DCP, and KV-Cache store
├── test_pipelined_and_numa.py # Unit tests for pipelined DMA and NUMA affinity
└── test_robustness.py # Thread/fork safety, framing, O_DIRECT bounce buffers, CRC verification
- Scans host block devices using sysfs (
/sys/block) and/dev. - Detects physical NVMe (
nvme*), virtual disks (vd*,xvd*), and SCSI/SATA (sd*). - Safety Filters: Automatically excludes the OS root partition drive, swap partitions, and active mounted filesystems to prevent data corruption.
- Mock fallback devices are strictly disabled by default (
auto_create_mock_devices = false).
- Manages POSIX shared memory chunks in
/dev/shm/hypercache/(with/tmp/hypercache/shm/fallback). - Delivers 50–100+ GB/s line-rate memory bus streaming for ephemeral AI workloads.
- Provides immediate drop-in compatibility for future CXL 2.0/3.0 Type 3 DAX appliances (
/dev/dax*). - Dynamically auto-sizes to 75% of
/dev/shmor configuredram_cache_capacity_bytes.
- When a model URI is requested (
locatewithauto_ingest=Trueorpreload):- Identifies backend from URI:
s3://orhypercache://s3/...routes to S3;file:///,nfs://, orhypercache://file/...routes to NFS. - Identifies target tier (
--tier ram,--tier nvme, or auto). - Divides data into fixed 4 MiB chunks (
chunk_size_bytes = 4194304). - Stripes chunks across POSIX shared memory or discovered NVMe drives in round-robin fashion.
- Records physical block offsets and device assignments in the Moka metadata manifest.
- Identifies backend from URI:
- Moka W-TinyLFU: High-throughput concurrent cache managing object metadata in RAM. Evicts least-frequently-used objects when combined storage capacity (NVMe + RAM) is reached.
- NVMe Persistence Manifest: Flushes metadata asynchronously to
/var/lib/hypercache/metadata.json(or fallback/tmp/hypercache/metadata.json). On daemon reboot, restores the entire cache catalog in <20 ms without re-reading raw disks.
- Reads raw block devices directly using
O_DIRECTorkvikio.CuFile(GPUDirect Storage), and reads shared memory chunks using zero-copymmap. - Bypasses the OS kernel page cache and CPU memory copies.
- Computes chunk offsets:
device_start_offset_bytes + (stripe_index * chunk_size_bytes).
- Controller Daemon (
./hypercache-admin controller):- Listens on
controller_bind_address:controller_port(default:0.0.0.0:9120). - Maintains
FleetRegistry: In-memory thread-safe map of active nodes, last heartbeat timestamps, drive lists, aggregate capacities, and cached footprints. - Automatically identifies remote node IP addresses from incoming TCP connections (
ConnectInfo). - Dispatches parallel HTTP requests (
futures_util::future::join_all) to target worker nodes (/api/v1/agent/preload,/api/v1/agent/evict,/api/v1/agent/purge).
- Listens on
- Worker Node Agent (
control-plane/src/agent_server.rs):- Sends heartbeats to the Controller every 5 seconds (
POST /api/v1/register). - Listens on port
9115for authenticated directives from the Controller. - Nodes that miss heartbeats for >15 seconds are marked
OFFLINE.
- Sends heartbeats to the Controller every 5 seconds (
- Local Loopback Zero-Config:
- Connections originating from
127.0.0.1/::1or the Controller host's bound IP bypass authentication, granting instant access to./hypercache-adminon the Controller node. - Controlled by
allow_local_admin_bypass(defaulttrue). It must befalsewherever a local proxy (K8s sidecar, nginx,kubectl port-forward) can forward remote traffic; the bundled K8s manifest sets it tofalse.
- Connections originating from
- Cluster Security Token (
cluster_auth_token/HYPERCACHE_AUTH_TOKEN):- When set, all remote network calls between the Controller and Worker Agents require
Authorization: Bearer <token>. - Node registrations with missing or invalid tokens are rejected with
HTTP 401 Unauthorized.
- When set, all remote network calls between the Controller and Worker Agents require
- Zero-Exposure Default Configuration:
- Single-node worker runs with
metrics_enabled = falseandagent_enabled = false, exposing no network ports. - The agent API refuses to bind a non-loopback address without
cluster_auth_token.
- Single-node worker runs with
- Local Data Access Confinement:
file:///nfs://URIs are confined tonfs_mount_pathandnfs_allowed_paths(canonicalized;..and symlink escapes rejected).- Synthetic data is never served unless
allow_synthetic_s3_fallback = trueor a request explicitly opts in for ahypercache://models/benchmark/URI.
# Using build script
./build.sh
# Or directly with Cargo
cargo build --release --manifest-path control-plane/Cargo.toml
cp control-plane/target/release/hypercache ./hypercache
cp control-plane/target/release/hypercache-admin ./hypercache-admincargo test --manifest-path control-plane/Cargo.tomlExpected: All unit and integration tests pass (70+ tests covering metadata, allocator, persistence, ingestion, RAM memory pool, discovery, auth, agent and controller security). The live S3 tests in tests/s3_live.rs are #[ignore]d and run by the verification harness.
.venv/bin/pytest client-py/testsExpected: All 34 client, datapath, fsspec and integration tests pass.
scripts/verify_release.sh # static, rust, python, e2e, s3, linux, image, k8s
scripts/verify_release.sh --quick # static, rust, pythonExpected: RESULT: PASSED. Docker-based stages (s3, linux, image, k8s) are skipped with a notice when Docker (or, for k8s, minikube and kubectl) is unavailable. Run the full harness before tagging a release.
# Start background daemon on mock or real drives (explicit config path required)
./hypercache -c configs/hypercache-agent.toml start &
DAEMON_PID=$!
# Run 5 GB benchmark
./hypercache bench --size-mb 5000
# Terminate test daemon
kill $DAEMON_PID- Never return freed extents straight to the free pool. Evicted or replaced chunks go to the quarantine in
metadata.rsand are reusable only afterreclaim_grace_secs. Readers resolve a layout over IPC and then read raw offsets with no lock, so early reuse makes them silently read another object's bytes. - Every layout mutation bumps
generation.locatereturns it. Write-back status updates must go throughupdate_sync_status_if_generationwith the generation they were started for, so an old flush cannot mark a newer commit as synced. - Go through single-flight ingest. Any code path that may ingest a URI (UDS
locate/ingest/preload, agent preload, batch) must callIngestionEngine::get_or_ingest/ingest_object_full, never a backend fetch directly. - Durable before visible. Chunk data is
fdatasynced and its CRC32 recorded beforesave_layoutpublishes it. Do not publish a layout that points at unsynced data. - One manifest writer. Mutations signal the coalescing writer in
metadata.rs. Do not writemetadata.jsonfrom anywhere else.atomic_write(unique tmp, fsync, rename, dir fsync) is the only correct way to replace it. - Dirty objects are never evictable.
COMMITTED,SYNCING_BACKENDandSYNC_FAILEDweigh 0 for eviction.resume_pending_writebacksre-queues them at boot. A newSyncStatusvariant must be classified inSyncStatus::is_dirty(protocol.rs). - Allocations roll back on failure. If striping fails partway, every extent allocated in that call is returned.
- New fields on IPC or manifest structs (
ChunkDescriptor,StoredMetadata,Response) must beOptionor#[serde(default)]. Manifests and Python clients from 0.9.0 must keep loading. - Enums on the wire use
#[serde(rename = "...")]with the existing strings ("ONLINE","SYNC_FAILED", ...). Do not rename them. - Add new TOML keys with defaults. Never rename or remove a key. Document each new key in
configs/*.tomland in the table in section 8. - The Python client must keep working against an older daemon. Treat missing response fields as optional.
- macOS hosts do not compile the Linux-only code (
#[cfg(target_os = "linux")]: sysfs discovery, O_DIRECT,fdatasyncpaths). Clippy and test failures there show up only on Linux. Runscripts/verify_release.sh --stages linux(Docker) before calling a change done. - Clippy depends on the toolchain. CI uses current stable. A local toolchain that is a few releases behind may miss new lints, so run
rustup updateif CI clippy fails but local clippy passes. - Containers expose host disks in sysfs without
/devnodes. Discovery must skip them (it logs at debug). Do not turn that case into an error. - Tests must not depend on the host's disks. GitHub runner hardware varies: some VMs expose an unmounted disk the non-root runner user cannot open, which
checkcorrectly reports as FAIL. Tests that call discovery orevaluate_hostmust pindevice_prefixes = [], mock devices or filesystem arenas. release.ymlruns the test suites without the harness. Any environment the harness sets up (for examplePYTHONPATH=client-pyfor pytest) must also be set in the workflow step.- Agents bound to loopback or
0.0.0.0advertise an unusable address. The controller dials them throughresolve_agent_address(falls back to the registration source IP, IPv6 bracketed). Use it for every new dispatch path. - Clients open cache files directly, outside the daemon's mount namespace. A layout carries the daemon's own paths, so a containerized daemon must share the socket directory, arena directories and (RAM tier) the shared-memory directory at identical paths with its clients. A container's
/dev/shmis private. Non-root clients needclient_gid. Never let the client create RAM chunk files or directories: the daemon pre-creates staged files andcommit_writerejects RAM chunks that were never written, which is how a client with a private/dev/shmis caught instead of silently losing a checkpoint.scripts/e2e/container_host_access.shandk8s_smoke.shcover this. - The daemon is PID 1 in containers. PID 1 ignores signals it has no handler for, so every long-running entry point must stop on
server::shutdown_signal()(SIGINT and SIGTERM), ordocker stopand pod deletion wait out the grace period and SIGKILL it. - Arenas are reserved at startup.
detect_devices_checked(used by the daemon) fails when an arena cannot be provisioned, and the daemon refuses to start.hypercache checkmust only inspect arenas (check_arena), never create or reserve them. Block-count (st_blocks) assertions are Linux-only: APFS reserves space without reporting it. - Raw-device and filesystem scenarios need root.
scripts/e2e/bare_metal_access.shruns in thee2estage only on Linux as root or with passwordless sudo (GitHub runners), and inside the privilegedlinuxstage container; elsewhere it is skipped with a notice. - Never hardcode the release version in tests or scripts. Read it from
control-plane/Cargo.toml(the static stage fails on a literal). A hardcoded0.9.1in the e2e script broke CI on the 0.9.2 bump. storage_backing_mode = "filesystem"must never touch raw disks. If no arena can be provisioned the daemon runs without Tier 2 storage. A privileged pod sees the host's unmounted disks, so a fallback scan would overwrite them.- Use
HYPERCACHE_SOCKET_PATHto isolate test daemons. It moves the socket and lock file for the daemon, the CLI and the Python client, so tests never touch/run/hypercacheor another running daemon. - MinIO images are no longer pullable. The live S3 tests use
versity/versitygw:v1.10.0.server_enforces_signaturesproves the server really checks SigV4. Keep it, or signing bugs pass silently. - Do not edit
scripts/verify_release.shwhile it is running. Bash reads the script incrementally, and an edit mid-run causes bogus syntax errors. - The zero-emoji scanner checks itself. Write emoji code point ranges as integers (
chr(0x1F300)), never as literal characters or\uescapes that a tool might decode. - Docker buildx on macOS can fail with a permission error on
~/.docker/buildx. Run the image stage withBUILDX_CONFIG=$(mktemp -d)instead of changing the user's Docker config.
| Change | Add a test in |
|---|---|
| Pure logic (allocator, selectors, auth, config parsing) | #[cfg(test)] module in the same .rs file |
| IPC behavior (request handling, commit, write-back) | server.rs tests (they run a real UDS round trip) |
| Anything a real S3 server must confirm (signing, pagination, multipart) | control-plane/tests/s3_live.rs (#[ignore], gated on HYPERCACHE_TEST_S3_*) |
| Multi-process behavior (restart, SIGKILL, eviction pressure, fleet dispatch, auth over HTTP) | a check in scripts/e2e/hypercache_e2e.py (scenario_daemon, scenario_fleet or scenario_s3) |
| Python client, datapath, integrations | client-py/tests/ (test_robustness.py for threading, fork, framing, O_DIRECT, CRC) |
A regression test should fail without the fix. When in doubt, revert the fix briefly and confirm that it does.
Run the E2E scenarios directly against release binaries:
cargo build --release --manifest-path control-plane/Cargo.toml
.venv/bin/python scripts/e2e/hypercache_e2e.py daemon fleet --keep # add s3 with --s3-endpoint ...Merging to main releases nothing. ci.yml only tests. Binaries, the container image and the PyPI package are built and published by release.yml, which runs when a v* tag is pushed. A fix on main without a tag has not reached users.
Versions are vMAJOR.MINOR.PATCH:
| Part | Bump it for | Example |
|---|---|---|
| MAJOR | The first stable release, and later breaking changes. Only the maintainer decides this. It stays 0 until then. | v0.9.4 to v1.0.0 |
| MINOR | A new user-facing feature, or a change that breaks the rules in section 7.B. Resets PATCH to 0. Describe any break and its upgrade steps in CHANGELOG.md. |
v0.9.4 to v0.10.0 |
| PATCH | Bug fixes, speed and hardening work that stay backward compatible, including a new optional config key that a fix needs. | v0.9.3 to v0.9.4 |
Rules:
- Every
v*tag publishes. That includes names likev0.9.4-rc1, so never push a tag to test something. - Tags are permanent. A ruleset blocks moving or deleting
v*tags, and PyPI never accepts a version number twice. If a release is bad, fix it in the next PATCH and never reuse the number. - Tag a commit on
main, after CI has passed on that commit. The tag must equal the version in the files listed below.release.ymlstops before publishing if it doesn't, but the tag and its number are still used up, so check first. - Release user-facing fixes soon. Wait only if another fix is about to land. Changes to docs, CI or tests alone don't need a release.
- Agents don't decide to release. When a user-facing change merges, tell the maintainer it is unreleased and ask whether to cut a release.
- Record changes as you make them. Add them under
## UnreleasedinCHANGELOG.md. The release renames that heading to the new version.
Release checklist:
- On a branch, bump the version in
control-plane/Cargo.toml,client-py/pyproject.toml,client-py/hypercache/__init__.pyand the default tag in.github/workflows/release.yml. The static stage fails if they disagree. The binaries read their version from Cargo (env!("CARGO_PKG_VERSION")), so never hardcode version strings in Rust. - Run
cargo updateonly on purpose.Cargo.lockis committed and every build uses--locked. - In
CHANGELOG.md, rename## Unreleasedto the new version and add a line saying what the release is for. - Run the full
scripts/verify_release.shuntil it printsRESULT: PASSED. - Open a PR. The maintainer reviews and merges it, because
mainonly accepts changes through a PR. - Once CI passes on that commit in
main, the maintainer tags it:git tag vX.Y.Z <commit> && git push origin vX.Y.Z. - Check that the
Release Binaries & Container Imagerun passed, then check the GitHub release, the container image and the PyPI package.
Leave committing, merging, tagging and pushing to the maintainer unless they explicitly ask you to do it.
8. Master Configuration Reference (configs/hypercache-agent.toml & configs/hypercache-controller.toml)
| Key | Type | Default | Description |
|---|---|---|---|
device_discovery_mode |
string | "auto" |
"auto" (scan physical/virtual drives) or "manual"
|
allow_virtual_drives |
bool | true |
Allow virtualized block devices (vd*, xvd*) |
include_partitions |
bool | true |
Include disk partitions (vdc1, nvme0n1p1) |
device_prefixes |
list | ["nvme", "vd", "xvd", "sd"] |
Allowed block device name prefixes |
exclude_root_device |
bool | true |
Exclude OS root disk from storage pool |
chunk_size_bytes |
integer | 4194304 |
Chunk striping size in bytes (4 MiB) |
device_start_offset_bytes |
integer | 4096 |
Safety offset from start of block device (4 KB) |
metadata_persistence_enabled |
bool | true |
Enable async NVMe metadata manifest sync |
metadata_persistence_path |
string | "/var/lib/hypercache/metadata.json" |
Path to NVMe metadata manifest |
ram_cache_enabled |
bool | false |
Enable Tier 1 Host RAM / CXL shared memory tier (off by default) |
ram_cache_capacity_bytes |
integer | 34359738368 |
RAM cache capacity in bytes (32 GiB default or 75% of /dev/shm) |
ram_cache_shm_path |
string | "/dev/shm/hypercache" |
POSIX shared memory directory path |
authoritative_backend |
string | "s3" |
Default backend: "s3" or "nfs"
|
s3_endpoint |
string | "http://127.0.0.1:9002" |
Authoritative S3 endpoint URL |
s3_region |
string | "us-east-1" |
S3 region |
s3_bucket |
string | "hypercache-authoritative" |
S3 bucket name |
nfs_mount_path |
string | "/mnt/nfs" |
POSIX/NFS root mount path |
metrics_enabled |
bool | false |
Enable Prometheus HTTP exporter (off by default) |
metrics_bind_address |
string | "127.0.0.1" |
IP to bind Prometheus metrics exporter on |
metrics_port |
integer | 9110 |
Port for Prometheus metrics exporter |
agent_enabled |
bool | false |
Enable Node Agent Fleet API (off by default) |
agent_bind_address |
string | "0.0.0.0" |
IP to bind Node Agent Fleet API on |
agent_http_port |
integer | 9115 |
Port for Node Agent Fleet API |
controller_url |
string | None |
URL of Central Controller for auto-registration |
controller_bind_address |
string | "0.0.0.0" |
IP to bind Central Controller service on |
controller_port |
integer | 9120 |
Port for Central Controller service |
cluster_auth_token |
string | None |
Shared secret Bearer token for cluster security |
node_id |
string | "$HOSTNAME" |
Unique identifier for the worker node |
node_labels |
map | {} |
Key-value labels for selective targeting |
storage_backing_mode |
string | "auto" |
"auto", "raw_block", or "filesystem" (O_DIRECT arena files for K8s/GDS) |
filesystem_cache_paths |
list | [] |
Mounted ext4/XFS directory paths for Direct I/O arena files |
filesystem_arena_size_bytes |
integer | 10737418240 |
Size per filesystem arena file in bytes (10 GiB default) |
allow_synthetic_s3_fallback |
bool | false |
Gate synthetic benchmark fallback on S3 error (false in production) |
peer_replication_factor |
integer | 1 |
Peer checkpoint replication factor (2 enables CommitWrite) |
p2p_preload_enabled |
bool | false |
Enable P2P fleet pre-warming (1 origin pull + peer chunk distribution) |
nfs_allowed_paths |
list | [] |
Extra roots file:// / nfs:// URIs may access (besides nfs_mount_path) |
reclaim_grace_secs |
integer | 30 |
Grace period before freed extents are reused (protects in-flight readers) |
s3_parallel_streams |
integer | 8 |
Parallel ranged GETs per object during S3 ingest |
s3_read_timeout_secs |
integer | 60 |
Per-read idle timeout for S3 transfers (no total-time cap) |
s3_session_token |
string | $AWS_SESSION_TOKEN |
Session token for temporary S3 credentials |
s3_multipart_threshold_bytes |
integer | 67108864 |
Uploads above this size use S3 multipart upload |
allow_local_admin_bypass |
bool | true |
Controller: loopback callers skip auth (set false behind proxies / in K8s) |
node_gc_secs |
integer | 600 |
Controller: forget nodes silent for this long |
client_gid |
integer | None |
Numeric group given read/write on the socket, arenas, RAM-tier files and claimed raw devices (non-root host / pod clients) |
filesystem_arena_preallocate |
bool | true |
Reserve arena space at startup (daemon refuses to start if it does not fit); false = sparse arenas |