diff --git a/docs/en/TOC.md b/docs/en/TOC.md index 2dc88c733aa..75773aa8511 100644 --- a/docs/en/TOC.md +++ b/docs/en/TOC.md @@ -23,6 +23,7 @@ - [Data Preloading](samples/data_warmup.md) - [CacheRuntime Data Operations](samples/cacheruntime/cacheruntime_data_operations.md) - [Use Curvine as CacheRuntime for Data Caching](samples/cacheruntime/curvine_cache_runtime.md) + - [Deploy Mooncake with CacheRuntime](samples/cacheruntime/mooncake_cache_runtime.md) - [Cache Runtime Manually Scaling](samples/dataset_scaling.md) - [Automatic Cleanup Data Operation](samples/automatic_clean_up_data_operation.md) + Security diff --git a/docs/en/dev/generic_cache_runtime_integration.md b/docs/en/dev/generic_cache_runtime_integration.md index 54f899458ed..3ec92dcbd0a 100644 --- a/docs/en/dev/generic_cache_runtime_integration.md +++ b/docs/en/dev/generic_cache_runtime_integration.md @@ -176,6 +176,17 @@ topology: imagePullPolicy: IfNotPresent ``` +#### Cache Systems Without a Client Component + +The master, worker and client components under `topology` are all optional in the API. You may omit the **client** component if the underlying cache system has both of the following characteristics: + +1. **No POSIX mount semantics**: the cache system itself provides no FUSE-like mount capability. +2. **Applications connect directly**: applications read and write against the master/worker through the cache system's own SDK or RPC client, without relying on a mount point. + +In that case the Master and Worker components start as usual, the Dataset reaches the Bound phase as usual, and cache status is still reported through the ReportSummary script. + +Note that Fluid still creates a PVC/PV for such a Dataset and reports them as Bound, but an application pod that mounts the PVC stays in ContainerCreating forever (reporting `timeout waiting for FUSE mount point`). For this kind of cache system, applications should talk to the cache service directly instead of going through the Dataset → PVC → volumeMounts path. For a complete configuration example, see [Deploy Mooncake with CacheRuntime](../samples/cacheruntime/mooncake_cache_runtime.md). + ### Step 2.5 User Creates Runtime ```yaml diff --git a/docs/en/samples/cacheruntime/mooncake_cache_runtime.md b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md new file mode 100644 index 00000000000..f93674f0126 --- /dev/null +++ b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md @@ -0,0 +1,477 @@ +# Example - Deploy Mooncake with CacheRuntime + +[Mooncake](https://github.com/kvcache-ai/Mooncake) is a distributed KVCache store built for LLM inference workloads. Unlike cache systems such as Alluxio or JuiceFS, Mooncake does not provide POSIX mount semantics: applications talk to the cache service directly through its own client library instead of reading and writing files through a mount point. + +Fluid's generic CacheRuntime supports this kind of cache system: the CacheRuntimeClass `topology` declares only the master and worker components, and omits the client component that would otherwise be responsible for mounting. This document walks through that minimal setup. + +For background on the client-less architecture, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## Prerequisites + +Before running this example, follow the [Installation Guide](../../userguide/install.md) to install Fluid, and verify that its components are running: + +```shell +$ kubectl get pod -n fluid-system +NAME READY STATUS RESTARTS AGE +cacheruntime-controller-58c775584-x9q47 1/1 Running 0 10m +csi-nodeplugin-fluid-74bpw 2/2 Running 0 10m +csi-nodeplugin-fluid-cjvc6 2/2 Running 0 10m +csi-nodeplugin-fluid-x4wwp 2/2 Running 0 10m +dataset-controller-58497b968b-q6ssv 1/1 Running 0 10m +fluid-webhook-74db64fdf4-7rq69 1/1 Running 0 10m +fluidapp-controller-7dbdc7696b-86pph 1/1 Running 0 10m +``` + +Typically you should see one pod each for `dataset-controller`, `cacheruntime-controller`, `fluid-webhook` and `fluidapp-controller`, plus one `csi-nodeplugin` pod per node. `cacheruntime-controller` is the one this example requires; it is enabled with `--set runtime.cacheruntime.enabled=true` when installing the Helm chart. + +> Note: this example requires a Fluid build that includes [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157). On earlier versions the controller panics when the client component is omitted from `topology`. + +### The demo image + +Fluid does not publish a Mooncake image. This example uses a demo image built from the Mooncake Python distribution plus two small scripts that Fluid invokes: + +| Path | Purpose | +|---|---| +| `/custom-entrypoint.sh` | Component entrypoint; starts the right process based on the role (master/worker) | +| `/reportSummary.sh` | Collects cache usage and emits it as JSON in the format Fluid expects | + +The build context for that image lives in this repository under [`samples/mooncake/docker`](../../../../samples/mooncake/docker), so you can build an equivalent image yourself: + +```shell +$ docker build -t /mooncake:v3 samples/mooncake/docker +$ docker push /mooncake:v3 +``` + +Note that `apt-get` and `pip` resolve to whatever versions are current at build time, so a fresh build produces a functionally equivalent image, not a bit-for-bit reproduction of the digest below. + +**Building your own image and substituting it in the manifests below is the recommended path.** For convenience, a prebuilt copy is also available, pinned to an immutable digest so this example keeps working even if the tag is moved: + +``` +btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` + +An Alibaba Cloud mirror is available for networks in mainland China. It is a copy of the same manifest, so it carries the identical digest: + +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` + +> Note: both registries are personal accounts belonging to the author of this example, not project-controlled infrastructure, and they carry no availability guarantee. Treat them as a convenience for trying the example out, and build from `samples/mooncake/docker` for anything beyond that. + +Neither script is specific to this image: you can build an equivalent image on top of any Mooncake distribution, as long as it provides the same two entry points. See the conventions in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## Running the Example + +### Create the CacheRuntimeClass + +**Review the CacheRuntimeClass to be created** + +```shell +$ cat<mooncake-cacheruntimeclass.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntimeClass +metadata: + name: mooncake-demo +fileSystemType: mooncakefs +topology: + master: + service: + headless: {} + executionEntries: + reportSummary: + command: + - bash + - -c + - /reportSummary.sh + timeout: 30 + template: + spec: + restartPolicy: Always + containers: + - name: master + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + command: + - /custom-entrypoint.sh + args: + - master + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50051 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50051 + name: rpc + - containerPort: 8080 + name: metadata + - containerPort: 9003 + name: metrics + worker: + service: + headless: {} + template: + spec: + restartPolicy: Always + containers: + - name: worker + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + command: + - /custom-entrypoint.sh + args: + - worker + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50052 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50052 + name: data + - containerPort: 9300 + name: http +EOF +``` + +`CacheRuntimeClass` is a CRD defined by Fluid that describes how a class of cache system runs on Kubernetes — the image each component uses, how it starts, which ports it exposes, and how Fluid collects its runtime status. + +A few things worth noting in this example: + +- Only `master` and `worker` are declared under `topology`. All three components (master, worker, client) are optional in the API, and Mooncake does not need a client component. +- Note that `fileSystemType` and `topology` are top-level fields — they do not live under `spec`. +- The master component exposes three ports: `50051` for RPC, `8080` for the metadata service, and `9003` for metrics. Applications connect directly to the first two. +- `reportSummary` points at `/reportSummary.sh` inside the image. Fluid executes it periodically in the component pod and writes the result to the Dataset's `status` field. For the required output format, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +**Create the CacheRuntimeClass** + +```shell +$ kubectl apply -f mooncake-cacheruntimeclass.yaml +cacheruntimeclass.data.fluid.io/mooncake-demo created + +$ kubectl get cacheruntimeclass +NAME AGE +mooncake-demo 0s +``` + +> Tip: if you modify the CacheRuntimeClass afterwards, you must delete and recreate the CacheRuntime for the change to take effect. A CacheRuntime renders the CacheRuntimeClass into workloads at creation time; later edits are not applied retroactively, and deleting the pods alone does not help. + +### Create the Dataset and CacheRuntime + +**Review the resources to be created** + +```shell +$ cat<mooncake-dataset-runtime.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: Dataset +metadata: + name: mooncake-demo + namespace: default +spec: + placement: Shared + accessModes: + - ReadWriteMany + mounts: + - name: mc + mountPoint: "mooncakefs:///" +--- +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntime +metadata: + name: mooncake-demo + namespace: default +spec: + runtimeClassName: mooncake-demo + master: + replicas: 1 + worker: + replicas: 2 + tieredStore: + levels: + - emptyDir: + quota: 1Gi + high: "0.8" + low: "0.5" +EOF +``` + +The `Dataset` describes the dataset itself, while the `CacheRuntime` describes the runtime instance providing cache service for it — here, one master replica and two worker replicas. + +About the `mounts` field: Mooncake does not mount any underlying storage (UFS); data is written directly into the cache by the client, so `mooncakefs:///` here is only a placeholder mount point. Fluid passes the contents of `mounts` through into the CacheRuntime config ConfigMap for the component containers to read, but because this example's CacheRuntimeClass declares no `mountUfs` execution entry, Fluid never performs an actual mount (see the notes on skipping MountUFS in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md)). In the API `mounts` is optional, but if you do declare it, it must contain at least one entry. + +**Create the resources** + +```shell +$ kubectl apply -f mooncake-dataset-runtime.yaml +dataset.data.fluid.io/mooncake-demo created +cacheruntime.data.fluid.io/mooncake-demo created +``` + +**Check the Dataset status** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo NotBound 6s +``` + +As shown above, `PHASE` is `NotBound`, meaning the `Dataset` is not yet bound to a cache runtime. Wait for the components to start: + +```shell +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-demo-master-0 1/1 Running 0 15s 10.244.2.13 fluid-mooncake-worker +mooncake-demo-worker-0 1/1 Running 0 15s 10.244.2.14 fluid-mooncake-worker +mooncake-demo-worker-1 1/1 Running 0 15s 10.244.1.7 fluid-mooncake-worker2 +``` + +**Check the Dataset status again** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo 2.00GiB 0B 2.00GiB 0.0 Bound 11s +``` + +`PHASE` is now `Bound` and the cache service is ready to use. + +> Note: as with other runtimes, Fluid also creates a PV and PVC for this Dataset, and they show as Bound. But since there is no client component, there is no FUSE mount point, so **application pods must not mount this PVC**. See [FAQ](#faq) at the end of this document. + +## Accessing the Cache + +Mooncake applications read and write data by connecting directly to the master service through its Python client, not through a mount point. + +**Review the application to be created** + +```shell +$ cat<mooncake-client.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-1 +spec: + nodeName: fluid-mooncake-worker2 + containers: + - name: client + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF +``` + +Note that this pod's `spec` has no `volumes` or `volumeMounts` — this is the main difference between this example and the other runtime examples. + +A few notes: + +- This example reuses the Mooncake image as the client environment since it already ships the Python client library. `command` is overridden with `sleep infinity` so the image's default entrypoint does not start it as a master or worker. +- `POD_IP` is injected via the downward API. The client's `local_hostname` must be the pod's own IP so that other nodes can connect back to fetch data. +- `nodeName` pins the pod to a specific node so that the cross-node read later is deterministic: the writer stays on one node and the reader on another. Replace `fluid-mooncake-worker2` with a real node name from your cluster (`kubectl get nodes`), and make sure it differs from the node used by client-2 below. + +**Start the application and write data** + +```shell +$ kubectl apply -f mooncake-client.yaml +$ kubectl exec -it mooncake-client-1 -- python3 +``` + +```python +import os, hashlib +from mooncake.store import MooncakeDistributedStore + +MASTER = "mooncake-demo-master-0.svc-mooncake-demo-master" + +store = MooncakeDistributedStore() +store.setup( + local_hostname=os.environ["POD_IP"], + metadata_server=f"http://{MASTER}:8080/metadata", + master_server_addr=f"{MASTER}:50051", + global_segment_size=0, + local_buffer_size=128 * 1024 * 1024, + protocol="tcp", + rdma_devices="", +) + +payload = os.urandom(4 * 1024 * 1024) +store.put("demo_key", payload) +got = store.get("demo_key") +print("put md5:", hashlib.md5(payload).hexdigest()) +print("get md5:", hashlib.md5(got).hexdigest()) +print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) +``` + +> The example connects through the master pod's stable DNS name `mooncake-demo-master-0.svc-mooncake-demo-master`, which pins the client to one specific replica. The Service name `svc-mooncake-demo-master` works just as well here: Fluid creates the component Service as headless (`clusterIP: None`) and declares no `ports` on it, but that only affects SRV records — the A record still resolves straight to the backing pod IPs, and the client connects to container ports `8080` and `50051` directly. The per-pod name is the safer habit because it stays pinned to a single replica if the master is ever scaled out. + +Output of `setup()` (key lines only): + +``` +I0815 05:30:15.076617 transfer_metadata_plugin.cpp:1293] Found active interface eth0 with IP 10.244.1.8 +I0815 05:30:15.077410 client_service.cpp:747] Transfer engine auto discovery is disabled for protocol: tcp +I0815 05:30:15.078089 real_client.cpp:734] Successfully created client on port 12699 after 1 attempt(s) +I0815 05:30:15.079909 real_client.cpp:767] Registering local memory: 134217728 bytes +I0815 05:30:15.080171 real_client.cpp:932] Global segment size is 0, skip mounting segment +0 +``` + +> With `global_segment_size` set to 0, the client log shows `Global segment size is 0, skip mounting segment`, meaning the client no longer allocates a local memory segment and all data is held by the Fluid-managed workers. +> +> The `http=404 body: metadata not found` message during startup is expected: the client is registering itself with the metadata service for the first time, so the corresponding key does not exist yet. + +Read/write result: + +``` +put md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +get md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +match: True +``` + +> Note: Mooncake's `put` silently skips a key that **already exists** — it does not overwrite the old data, and it still returns `0`, so the return value gives nothing away. If you run the snippet above a second time (or also run a write in client-2 later on), the second `put` has no effect and `get` returns the content written the first time, producing `match: False`. This is not a cache failure. Before re-running this example, use a different key name, or delete and recreate the CacheRuntime so the workers' cache is cleared. + +**Reading across pods and nodes** + +First exit the Python session in the first client so the writer process terminates: + +```python +>>> exit() +``` + +Then create a second client pod, with `nodeName` pointing at a **different** node (client-1 is on `fluid-mooncake-worker2`, so this one uses `fluid-mooncake-worker`): + +```shell +$ cat<mooncake-client-2.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-2 +spec: + nodeName: fluid-mooncake-worker + containers: + - name: client + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF + +$ kubectl apply -f mooncake-client-2.yaml +$ kubectl get pod mooncake-client-1 mooncake-client-2 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-mooncake-worker2 +mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker +``` + +The second pod runs a brand-new Python process, so open a session in it and repeat the imports and the full `MooncakeDistributedStore` initialization shown above — the `setup` parameters are identical, and `POD_IP` again resolves to this pod's own IP: + +```shell +$ kubectl exec -it mooncake-client-2 -- python3 +``` + +Once `store` and `hashlib` are initialized, read and verify: + +```python +got = store.get("demo_key") +print("len:", len(got)) +print("md5:", hashlib.md5(got).hexdigest()) +``` + +Output: + +``` +len: 4194304 +md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +``` + +The md5 matches the writer exactly. Note that by this point the client process that wrote the data has already exited, and the reader is on a different node — showing that the cached data is held by the Fluid-managed workers, depending neither on the client process that wrote it nor on the node it ran on. + +## Inspecting Cache Status + +```shell +$ kubectl get dataset mooncake-demo -o yaml +``` + +> Note: Fluid runs the ReportSummary script periodically, so the Dataset `status` does not update the moment a write completes — in practice it takes about a minute to refresh. Until then you will still see `cached: 0B` and `fileNum: "0"`, which is expected; just check again shortly. To see the cache system's live state directly, run the script inside the master pod: +> +> ```shell +> $ kubectl exec mooncake-demo-master-0 -- bash -c /reportSummary.sh +> ``` + +```yaml + cacheStates: + cacheCapacity: 2.00GiB + cacheHitRatio: "0" + cached: 4.00MiB + cachedPercentage: "0.2" + fileNum: "1" + ufsTotal: 2.00GiB + conditions: + - lastTransitionTime: "2026-08-15T05:28:03Z" + lastUpdateTime: "2026-08-15T05:28:03Z" + message: The ddc runtime is ready. + reason: DatasetReady + status: "True" + type: Ready +``` + +`cached: 4.00MiB` matches the 4 MiB written above, and `fileNum: 1` corresponds to the single key. `cacheCapacity: 2.00GiB` comes from the 1Gi `tieredStore` quota on each of the two workers. + +The remaining two fields need to be read in the context of having no UFS: + +- `ufsTotal` normally reports the total data size in the underlying storage. Mooncake has no UFS, so the example image's `reportSummary.sh` fills the field with the total cache capacity instead (`UFS_TOTAL="$CACHE_CAPACITY"` in the script), which is why it equals `cacheCapacity`. This is a choice made by the script, not an anomaly. +- `cacheHitRatio` is approximated by the script from the master's `Get` request success rate. Note that `Requests (Success/Total per sec)` in the master's `/metrics/summary` is a **per-second instantaneous rate**, not a cumulative counter, so if no Get is in flight when the script samples, it reads `0.00/0.00` — which is why this field usually shows `0`. It reflects request success rate at the sampling instant, not a strict cache hit ratio. + +All of these values are collected and reported by the ReportSummary script configured in the CacheRuntimeClass; Fluid only passes them through. For the meaning of each field and the required output format, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## FAQ + +### An application pod is stuck in ContainerCreating + +**Symptom**: an application pod that mounts this Dataset's PVC never starts, and its events show `FailedMount`. + +```shell +$ kubectl describe pod +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 60s default-scheduler Successfully assigned default/ to fluid-mooncake-worker + Warning FailedMount 29s kubelet MountVolume.SetUp failed for volume "default-mooncake-demo" : rpc error: code = Internal desc = timeout waiting for FUSE mount point to be ready +``` + +**Cause**: this example's CacheRuntimeClass declares no client component, so there is no FUSE mount point. The CSI plugin waits for a mount point that never appears until it times out, which produces the `timeout waiting for FUSE mount point to be ready` above. Fluid still creates the PVC/PV and reports them as Bound, but application pods cannot consume them through `volumeMounts`. + +**Resolution**: remove the PVC mount from the application pod, connect to the cache service directly as shown above, and delete the stuck pod. + +```shell +$ kubectl delete pod +``` + +## Cleanup + +```shell +$ kubectl delete -f mooncake-client-2.yaml +$ kubectl delete -f mooncake-client.yaml +$ kubectl delete -f mooncake-dataset-runtime.yaml +$ kubectl delete -f mooncake-cacheruntimeclass.yaml +``` diff --git a/docs/zh/TOC.md b/docs/zh/TOC.md index d00a1825043..eab87e5585c 100644 --- a/docs/zh/TOC.md +++ b/docs/zh/TOC.md @@ -28,6 +28,7 @@ - [数据预加载](samples/data_warmup.md) - [CacheRuntime 数据操作](samples/cacheruntime/cacheruntime_data_operations.md) - [使用 Curvine 作为 CacheRuntime 进行数据缓存](samples/cacheruntime/curvine_cache_runtime.md) + - [使用 CacheRuntime 部署 Mooncake](samples/cacheruntime/mooncake_cache_runtime.md) - [Cache Runtime手动扩缩容](samples/dataset_scaling.md) - [数据操作自动清理](samples/automatic_clean_up_data_operation.md) + 安全 diff --git a/docs/zh/dev/generic_cache_runtime_integration.md b/docs/zh/dev/generic_cache_runtime_integration.md index 14d666e890c..6c63f121889 100644 --- a/docs/zh/dev/generic_cache_runtime_integration.md +++ b/docs/zh/dev/generic_cache_runtime_integration.md @@ -175,6 +175,17 @@ topology: - custom-endpoint.sh imagePullPolicy: IfNotPresent ``` +#### 无 Client 组件的场景 + +topology 中的 master、worker、client 三个组件在 API 上均为可选。如果底层缓存系统具备以下特征，可以**不声明 client**： + +1. **无 POSIX 挂载语义**：缓存系统本身不提供 FUSE 之类的挂载能力 +2. **应用直连访问**：应用通过缓存系统自带的 SDK 或 RPC 客户端直接读写 master/worker，不依赖挂载点 + +此时 Master 与 Worker 组件正常拉起，Dataset 正常进入 Bound 状态，缓存状态也会通过 ReportSummary 脚本正常上报。 + +需要注意的是，Fluid 仍会为该 Dataset 创建 PVC/PV 且状态为 Bound，但业务 Pod 挂载该 PVC 会一直停留在 ContainerCreating（报 timeout waiting for FUSE mount point）。这类缓存系统应由应用直连缓存服务读写，不要走 Dataset → PVC → volumeMounts 这条路径。完整配置示例参见 [使用 CacheRuntime 部署 Mooncake](../samples/cacheruntime/mooncake_cache_runtime.md)。 + ### 步骤2.5 用户创建Runtime diff --git a/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md new file mode 100644 index 00000000000..326ec1803ea --- /dev/null +++ b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md @@ -0,0 +1,477 @@ +# 示例 - 使用 CacheRuntime 部署 Mooncake + +[Mooncake](https://github.com/kvcache-ai/Mooncake) 是一个面向大模型推理场景的分布式 KVCache 存储系统。与 Alluxio、JuiceFS 等缓存系统不同，Mooncake 不提供 POSIX 挂载语义，应用通过其自带的客户端直接与缓存服务通信，而不是通过挂载点读写文件。 + +Fluid 的通用 CacheRuntime 支持这类缓存系统：在 CacheRuntimeClass 的 `topology` 中只声明 master 和 worker 组件，不声明负责挂载的 client 组件。本文档演示这一最小实现。 + +关于无 client 架构的说明，参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 前提条件 + +在运行该示例之前，请参考[安装文档](../../userguide/install.md)完成安装，并检查 Fluid 各组件正常运行： + +```shell +$ kubectl get pod -n fluid-system +NAME READY STATUS RESTARTS AGE +cacheruntime-controller-58c775584-x9q47 1/1 Running 0 10m +csi-nodeplugin-fluid-74bpw 2/2 Running 0 10m +csi-nodeplugin-fluid-cjvc6 2/2 Running 0 10m +csi-nodeplugin-fluid-x4wwp 2/2 Running 0 10m +dataset-controller-58497b968b-q6ssv 1/1 Running 0 10m +fluid-webhook-74db64fdf4-7rq69 1/1 Running 0 10m +fluidapp-controller-7dbdc7696b-86pph 1/1 Running 0 10m +``` + +通常来说，你会看到 `dataset-controller`、`cacheruntime-controller`、`fluid-webhook`、`fluidapp-controller` 各一个 Pod，以及每个节点一个的 `csi-nodeplugin` Pod 正在运行。其中 `cacheruntime-controller` 是本示例必需的，它由 Helm 安装时的 `--set runtime.cacheruntime.enabled=true` 开启。 + +> 注意：本示例需要包含 [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157) 的 Fluid 版本。较早版本在 `topology` 中省略 client 组件时 controller 会发生 panic。 + +### 示例镜像 + +Fluid 并不发布 Mooncake 镜像。本示例使用的是一个演示镜像，它在 Mooncake 的 Python 发行版之上加入了两个供 Fluid 调用的脚本： + +| 路径 | 作用 | +|---|---| +| `/custom-entrypoint.sh` | 组件启动入口，按角色（master/worker）启动对应进程 | +| `/reportSummary.sh` | 采集缓存用量并按 Fluid 要求的 JSON 格式输出 | + +该镜像的构建上下文已随本仓库提供，位于 [`samples/mooncake/docker`](../../../../samples/mooncake/docker)，你可以据此构建等价镜像： + +```shell +$ docker build -t /mooncake:v3 samples/mooncake/docker +$ docker push /mooncake:v3 +``` + +注意 `apt-get` 和 `pip` 在构建时拉取的都是当时的最新版本，因此重新构建得到的是功能等价的镜像，并不能字节级复现下面的摘要。 + +**推荐的做法是自行构建镜像，并替换下文各处清单中的镜像地址。** 为方便试用，也提供了一份预构建镜像，并固定到不可变的镜像摘要，这样即使标签被覆盖，本示例仍然可用： + +``` +btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` + +国内网络环境下也可使用阿里云镜像。它是同一 manifest 的副本，摘要与上面完全相同： + +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` + +> 注意：以上两个仓库均为本示例作者的个人账号，并非项目管控的基础设施，不提供任何可用性保证。它们仅用于快速试用本示例；正式使用请基于 `samples/mooncake/docker` 自行构建。 + +这两个脚本并非该镜像独有：你也可以基于任意 Mooncake 发行版自行构建等价镜像，只要它提供同样的两个入口即可，具体约定参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 运行示例 + +### 创建 CacheRuntimeClass + +**查看待创建的 CacheRuntimeClass 资源对象** + +```shell +$ cat<mooncake-cacheruntimeclass.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntimeClass +metadata: + name: mooncake-demo +fileSystemType: mooncakefs +topology: + master: + service: + headless: {} + executionEntries: + reportSummary: + command: + - bash + - -c + - /reportSummary.sh + timeout: 30 + template: + spec: + restartPolicy: Always + containers: + - name: master + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + command: + - /custom-entrypoint.sh + args: + - master + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50051 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50051 + name: rpc + - containerPort: 8080 + name: metadata + - containerPort: 9003 + name: metrics + worker: + service: + headless: {} + template: + spec: + restartPolicy: Always + containers: + - name: worker + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + command: + - /custom-entrypoint.sh + args: + - worker + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50052 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50052 + name: data + - containerPort: 9300 + name: http +EOF +``` + +`CacheRuntimeClass` 是 Fluid 定义的 CRD，用于描述一类缓存系统如何在 Kubernetes 上运行——包括各组件使用的镜像、启动方式、端口，以及 Fluid 如何采集其运行状态。 + +本示例中需要留意的几点： + +- `topology` 下只声明了 `master` 和 `worker`。master、worker、client 三个组件在 API 中均为可选字段，Mooncake 无需 client 组件。 +- 注意 `fileSystemType` 和 `topology` 是顶层字段，不在 `spec` 之下。 +- master 组件暴露三个端口：`50051` 用于 RPC，`8080` 用于元数据服务，`9003` 用于 metrics。应用将直接连接前两个。 +- `reportSummary` 指向镜像中的 `/reportSummary.sh`，Fluid 会周期性地在组件 Pod 中执行它，并将结果更新到 Dataset 的 `status` 字段。其输出格式要求参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +**创建 CacheRuntimeClass 资源对象** + +```shell +$ kubectl apply -f mooncake-cacheruntimeclass.yaml +cacheruntimeclass.data.fluid.io/mooncake-demo created + +$ kubectl get cacheruntimeclass +NAME AGE +mooncake-demo 0s +``` + +> 提示：如果之后修改了 CacheRuntimeClass，需要删除并重建 CacheRuntime 才会生效。CacheRuntime 在创建时就已将当时的 CacheRuntimeClass 渲染为工作负载，后续修改不会回溯更新，仅删除 Pod 也无效。 + +### 创建 Dataset 与 CacheRuntime + +**查看待创建的资源对象** + +```shell +$ cat<mooncake-dataset-runtime.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: Dataset +metadata: + name: mooncake-demo + namespace: default +spec: + placement: Shared + accessModes: + - ReadWriteMany + mounts: + - name: mc + mountPoint: "mooncakefs:///" +--- +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntime +metadata: + name: mooncake-demo + namespace: default +spec: + runtimeClassName: mooncake-demo + master: + replicas: 1 + worker: + replicas: 2 + tieredStore: + levels: + - emptyDir: + quota: 1Gi + high: "0.8" + low: "0.5" +EOF +``` + +`Dataset` 描述数据集本身，`CacheRuntime` 描述为该数据集提供缓存服务的运行时实例——本示例中即一个 master 副本和两个 worker 副本。 + +关于 `mounts` 字段：Mooncake 不挂载任何底层存储（UFS），数据由客户端直接写入缓存，因此这里的 `mooncakefs:///` 只是一个占位挂载点。Fluid 会把 `mounts` 的内容透传到 CacheRuntime 的配置 ConfigMap 中，供组件容器读取；但由于本示例的 CacheRuntimeClass 未声明 `mountUfs` 执行入口，Fluid 不会真正执行任何挂载动作（详见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)中关于"可以不设置 MountUFS"的说明）。API 上 `mounts` 是可选字段，但一旦声明就至少需要一项。 + +**创建资源对象** + +```shell +$ kubectl apply -f mooncake-dataset-runtime.yaml +dataset.data.fluid.io/mooncake-demo created +cacheruntime.data.fluid.io/mooncake-demo created +``` + +**查看 Dataset 资源对象状态** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo NotBound 6s +``` + +如上所示，`PHASE` 属性值为 `NotBound`，这意味着该 `Dataset` 资源对象目前还未与缓存运行时绑定。等待各组件启动： + +```shell +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-demo-master-0 1/1 Running 0 15s 10.244.2.13 fluid-mooncake-worker +mooncake-demo-worker-0 1/1 Running 0 15s 10.244.2.14 fluid-mooncake-worker +mooncake-demo-worker-1 1/1 Running 0 15s 10.244.1.7 fluid-mooncake-worker2 +``` + +**再次查看 Dataset 资源对象状态** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo 2.00GiB 0B 2.00GiB 0.0 Bound 11s +``` + +此时 `PHASE` 已变为 `Bound`，缓存服务可以使用了。 + +> 注意：与其他 Runtime 一样，Fluid 也会为该 Dataset 创建 PV 和 PVC，且状态显示为 Bound。但由于没有 client 组件，不存在 FUSE 挂载点，**业务 Pod 不应挂载该 PVC**。详见文末[常见问题](#常见问题)。 + +## 访问缓存 + +Mooncake 的应用通过其 Python 客户端直连 master 服务读写数据，不经过挂载点。 + +**查看待创建的应用** + +```shell +$ cat<mooncake-client.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-1 +spec: + nodeName: fluid-mooncake-worker2 + containers: + - name: client + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF +``` + +注意该 Pod 的 `spec` 中没有 `volumes` 和 `volumeMounts`——这是本示例与其他 Runtime 示例最主要的区别。 + +几点说明： + +- 本示例直接复用 Mooncake 镜像作为客户端环境，它已包含 Python 客户端库；`command` 覆盖为 `sleep infinity`，避免镜像的默认入口把它启动成 master 或 worker。 +- `POD_IP` 通过 downward API 注入，客户端的 `local_hostname` 必须是 Pod 自身 IP，其他节点才能回连取数。 +- `nodeName` 显式指定节点，是为了让后面的跨节点读取有确定的效果：写入方固定在一个节点，读取方固定在另一个节点。请把 `fluid-mooncake-worker2` 换成你集群中的实际节点名（`kubectl get nodes`），并确保与后面 client-2 使用的节点不同。 + +**启动应用并写入数据** + +```shell +$ kubectl apply -f mooncake-client.yaml +$ kubectl exec -it mooncake-client-1 -- python3 +``` + +```python +import os, hashlib +from mooncake.store import MooncakeDistributedStore + +MASTER = "mooncake-demo-master-0.svc-mooncake-demo-master" + +store = MooncakeDistributedStore() +store.setup( + local_hostname=os.environ["POD_IP"], + metadata_server=f"http://{MASTER}:8080/metadata", + master_server_addr=f"{MASTER}:50051", + global_segment_size=0, + local_buffer_size=128 * 1024 * 1024, + protocol="tcp", + rdma_devices="", +) + +payload = os.urandom(4 * 1024 * 1024) +store.put("demo_key", payload) +got = store.get("demo_key") +print("put md5:", hashlib.md5(payload).hexdigest()) +print("get md5:", hashlib.md5(got).hexdigest()) +print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) +``` + +> 本示例使用 master Pod 的稳定 DNS 名 `mooncake-demo-master-0.svc-mooncake-demo-master` 作为连接地址，从而固定访问某一个副本。这里用 Service 名 `svc-mooncake-demo-master` 同样可行：Fluid 创建的组件 Service 是 headless 的（`clusterIP: None`）且未声明 `ports`，但这只影响 SRV 记录，A 记录仍会直接解析到后端 Pod IP，客户端随后直连容器的 `8080` 和 `50051` 端口。之所以推荐使用 Pod 级 DNS 名，是因为它在 master 扩容后仍然固定指向单个副本。 + +`setup()` 的输出（截取关键部分）： + +``` +I0815 05:30:15.076617 transfer_metadata_plugin.cpp:1293] Found active interface eth0 with IP 10.244.1.8 +I0815 05:30:15.077410 client_service.cpp:747] Transfer engine auto discovery is disabled for protocol: tcp +I0815 05:30:15.078089 real_client.cpp:734] Successfully created client on port 12699 after 1 attempt(s) +I0815 05:30:15.079909 real_client.cpp:767] Registering local memory: 134217728 bytes +I0815 05:30:15.080171 real_client.cpp:932] Global segment size is 0, skip mounting segment +0 +``` + +> `global_segment_size` 设为 0 时，客户端日志中会出现 `Global segment size is 0, skip mounting segment`，表示客户端不再分配本地内存段，数据全部由 Fluid 管理的 worker 承载。 +> +> 启动过程中出现的 `http=404 body: metadata not found` 属于正常现象：客户端首次向元数据服务注册自身时，对应的 key 尚不存在。 + +读写结果： + +``` +put md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +get md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +match: True +``` + +> 注意：Mooncake 的 `put` 对**已存在**的 key 是静默跳过的——不覆盖旧数据，且仍然返回 `0`，从返回值上看不出区别。因此如果你重复执行上面这段代码（或在后面的 client-2 里也执行了写入），第二次的 `put` 不会生效，`get` 读回的仍是第一次写入的内容，于是出现 `match: False`。这不是缓存故障。重跑本示例前，请换一个 key 名，或删除并重建 CacheRuntime 让 worker 的缓存清空。 + +**跨 Pod 跨节点读取** + +先退出第一个客户端的 Python 交互环境，让写入方进程结束： + +```python +>>> exit() +``` + +再创建第二个客户端 Pod，`nodeName` 指向**另一个**节点（client-1 在 `fluid-mooncake-worker2`，这里用 `fluid-mooncake-worker`）： + +```shell +$ cat<mooncake-client-2.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-2 +spec: + nodeName: fluid-mooncake-worker + containers: + - name: client + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF + +$ kubectl apply -f mooncake-client-2.yaml +$ kubectl get pod mooncake-client-1 mooncake-client-2 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-mooncake-worker2 +mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker +``` + +第二个 Pod 中是一个全新的 Python 进程，需要先进入它的 Python 环境，并重复上文的导入语句和完整的 `MooncakeDistributedStore` 初始化（`setup` 参数与上文完全相同，`POD_IP` 同样取当前 Pod 自身的 IP）： + +```shell +$ kubectl exec -it mooncake-client-2 -- python3 +``` + +`store` 和 `hashlib` 初始化完成后，再做读取和校验： + +```python +got = store.get("demo_key") +print("len:", len(got)) +print("md5:", hashlib.md5(got).hexdigest()) +``` + +输出： + +``` +len: 4194304 +md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +``` + +md5 与写入方完全一致。需要强调的是，此时写入数据的那个客户端进程已经退出，且读取方位于另一个节点上——这说明缓存数据由 Fluid 管理的 worker 承载，既不依赖写入它的客户端进程，也不依赖所在节点。 + +## 查看缓存状态 + +```shell +$ kubectl get dataset mooncake-demo -o yaml +``` + +> 注意：Fluid 是周期性执行 ReportSummary 脚本的，写入完成后 Dataset 的 `status` 不会立刻更新。实测约需 1 分钟左右才会刷新，在此之前看到的仍是 `cached: 0B`、`fileNum: "0"`，属于正常现象，稍等再查即可。若想确认缓存系统侧的即时状态，可以直接在 master Pod 中执行该脚本： +> +> ```shell +> $ kubectl exec mooncake-demo-master-0 -- bash -c /reportSummary.sh +> ``` + +```yaml + cacheStates: + cacheCapacity: 2.00GiB + cacheHitRatio: "0" + cached: 4.00MiB + cachedPercentage: "0.2" + fileNum: "1" + ufsTotal: 2.00GiB + conditions: + - lastTransitionTime: "2026-08-15T05:28:03Z" + lastUpdateTime: "2026-08-15T05:28:03Z" + message: The ddc runtime is ready. + reason: DatasetReady + status: "True" + type: Ready +``` + +`cached: 4.00MiB` 与上文写入的 4 MiB 数据一致，`fileNum: 1` 对应写入的一个 key。`cacheCapacity: 2.00GiB` 则来自两个 worker 各 1Gi 的 `tieredStore` 配额。 + +其余两个字段需要结合无 UFS 的场景理解： + +- `ufsTotal` 在其他 Runtime 中表示底层存储的数据总量。Mooncake 没有 UFS，示例镜像的 `reportSummary.sh` 直接用缓存总容量填充该字段（脚本中即 `UFS_TOTAL="$CACHE_CAPACITY"`），因此它与 `cacheCapacity` 相等。这是脚本的上报选择，不是异常。 +- `cacheHitRatio` 由脚本从 master 的 `Get` 请求成功率近似估算。注意 master `/metrics/summary` 中的 `Requests (Success/Total per sec)` 是**每秒瞬时速率**而非累计计数，采样时若没有正在进行的 Get 请求，读到的就是 `0.00/0.00`，因此该字段通常显示为 `0`。它反映的是采样瞬间的请求成功率，并不是严格意义上的缓存命中率。 + +这些数据均由 CacheRuntimeClass 中配置的 ReportSummary 脚本采集上报，Fluid 只做透传，各字段的语义和输出格式要求参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 常见问题 + +### 业务 Pod 一直处于 ContainerCreating + +**现象**：业务 Pod 挂载了该 Dataset 对应的 PVC 后一直无法启动，事件中出现 `FailedMount`。 + +```shell +$ kubectl describe pod +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 60s default-scheduler Successfully assigned default/ to fluid-mooncake-worker + Warning FailedMount 29s kubelet MountVolume.SetUp failed for volume "default-mooncake-demo" : rpc error: code = Internal desc = timeout waiting for FUSE mount point to be ready +``` + +**原因**：本示例的 CacheRuntimeClass 未声明 client 组件，因此不存在 FUSE 挂载点。CSI 插件会一直等待挂载点就绪直到超时，于是报出上面的 `timeout waiting for FUSE mount point to be ready`。虽然 Fluid 仍会创建 PVC/PV 且状态为 Bound，但业务 Pod 无法通过 `volumeMounts` 使用它。 + +**处理**：从业务 Pod 中移除该 PVC 的挂载，改为按上文方式直连缓存服务，然后删除卡住的 Pod。 + +```shell +$ kubectl delete pod +``` + +## 环境清理 + +```shell +$ kubectl delete -f mooncake-client-2.yaml +$ kubectl delete -f mooncake-client.yaml +$ kubectl delete -f mooncake-dataset-runtime.yaml +$ kubectl delete -f mooncake-cacheruntimeclass.yaml +``` diff --git a/samples/mooncake/docker/Dockerfile b/samples/mooncake/docker/Dockerfile new file mode 100644 index 00000000000..97ace579e19 --- /dev/null +++ b/samples/mooncake/docker/Dockerfile @@ -0,0 +1,35 @@ +# Build context for the image used by the Mooncake CacheRuntime sample. +# +# See docs/en/samples/cacheruntime/mooncake_cache_runtime.md +# docs/zh/samples/cacheruntime/mooncake_cache_runtime.md +# +# Build and push to a registry you control: +# docker build -t /mooncake:v3 samples/mooncake/docker +# docker push /mooncake:v3 + +FROM python:3.12-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2 curl jq \ + && rm -rf /var/lib/apt/lists/* + +# Install from wheels only (--only-binary) so no package build/setup script is +# executed at install time, and pin every resolved version explicitly. +RUN pip install --no-cache-dir --only-binary=:all: \ + mooncake-transfer-engine-non-cuda==0.3.12.post1 \ + nvidia-cuda-runtime-cu12==12.8.90 + +ENV LD_LIBRARY_PATH=/usr/local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH + +# master rpc / worker data / metadata / master metrics / worker http +EXPOSE 50051 50052 8080 9003 9300 + +COPY custom-entrypoint.sh /custom-entrypoint.sh +COPY reportSummary.sh /reportSummary.sh +RUN chmod +x /custom-entrypoint.sh /reportSummary.sh + +# Run the components as an unprivileged user. All listening ports are above +# 1024 and the sample writes nothing to the container filesystem, so no root +# capability is required at runtime. +RUN useradd --create-home --uid 10001 --user-group mooncake +USER 10001 diff --git a/samples/mooncake/docker/custom-entrypoint.sh b/samples/mooncake/docker/custom-entrypoint.sh new file mode 100755 index 00000000000..ae157867750 --- /dev/null +++ b/samples/mooncake/docker/custom-entrypoint.sh @@ -0,0 +1,81 @@ +#!/bin/sh +# Component entrypoint invoked by Fluid's CacheRuntime. +# Usage: /custom-entrypoint.sh start + +set -e + +ROLE="$1" +ACTION="$2" + +if [ "$ACTION" != "start" ]; then + echo "Error: unsupported action '$ACTION'" + exit 1 +fi + +case "$ROLE" in + + master) + exec mooncake_master \ + -v=1 \ + --rpc_interface=eth0 \ + --enable_http_metadata_server=true \ + --http_metadata_server_host=0.0.0.0 \ + --http_metadata_server_port=8080 \ + --enable_metadata_cleanup_on_timeout=true \ + --client_ttl=10 + ;; + + worker) + # Read the runtime config JSON that Fluid mounts into the component pod. + if [ -z "$FLUID_RUNTIME_CONFIG_PATH" ] || [ ! -f "$FLUID_RUNTIME_CONFIG_PATH" ]; then + echo "Error: FLUID_RUNTIME_CONFIG_PATH not set or file not found" + exit 1 + fi + + CONFIG=$(cat "$FLUID_RUNTIME_CONFIG_PATH") + + MASTER_SVC=$(echo "$CONFIG" | jq -r '.master.service.name') + WORKER_SVC=$(echo "$CONFIG" | jq -r '.worker.service.name') + QUOTA=$(echo "$CONFIG" | jq -r '.worker.tieredStoreLevels[0].quotas[0] // "1GiB"') + + # Fluid reports the quota in Kubernetes units ("1Gi"); Mooncake expects "1GB". + SEGMENT_SIZE=$(echo "$QUOTA" | sed 's/Gi$/GB/; s/Mi$/MB/') + + NAMESPACE="${FLUID_DATASET_NAMESPACE:-default}" + MASTER_ADDR="${MASTER_SVC}.${NAMESPACE}.svc.cluster.local:50051" + METADATA_ADDR="http://${MASTER_SVC}.${NAMESPACE}.svc.cluster.local:8080/metadata" + WORKER_HOST="${POD_NAME}.${WORKER_SVC}.${NAMESPACE}.svc.cluster.local" + + echo "Starting worker: master=$MASTER_ADDR, segment_size=$SEGMENT_SIZE, host=$WORKER_HOST" + + # The metadata endpoint is plain HTTP and the transfer protocol is TCP + # because Mooncake exposes no TLS variant for either: the master's built-in + # metadata server (--enable_http_metadata_server) only speaks HTTP, and the + # transfer engine only offers "tcp" and "rdma". Both connections stay inside + # the cluster, addressed by ClusterIP/headless service DNS, and carry cache + # blocks between components of this runtime only. Put the runtime in a + # dedicated namespace with a NetworkPolicy, or a service mesh with mTLS, if + # that traffic needs to be protected on the wire. + exec mooncake_client \ + --host="$WORKER_HOST" \ + --port=50052 \ + --global_segment_size="$SEGMENT_SIZE" \ + --master_server_address="$MASTER_ADDR" \ + --metadata_server="$METADATA_ADDR" \ + --protocol=tcp \ + --enable_http_server=true \ + --http_port=9300 + ;; + + client) + # Mooncake is a client-less cache system in Fluid: applications link the + # Mooncake client library directly, so no client component is deployed. + echo "Error: client role is not applicable to Mooncake" + exit 1 + ;; + + *) + echo "Error: unknown role '$ROLE'" + exit 1 + ;; +esac diff --git a/samples/mooncake/docker/reportSummary.sh b/samples/mooncake/docker/reportSummary.sh new file mode 100755 index 00000000000..f360f7d47bf --- /dev/null +++ b/samples/mooncake/docker/reportSummary.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# ReportSummary entrypoint invoked by Fluid's CacheRuntime. It samples the +# Mooncake master's metrics endpoint and prints the JSON that Fluid writes into +# the Dataset's status.cacheStates field. +set -euo pipefail + +RAW=$(curl -s http://localhost:9003/metrics/summary) + +if [ -z "$RAW" ]; then + echo "Error: empty response from metrics endpoint" >&2 + exit 1 +fi + +# Extract the "Mem Storage: 0 B / 2.00 GB (0.0%)" segment. +MEM_LINE=$(echo "$RAW" | grep -oE 'Mem Storage: [^|]+' || true) + +CACHED_RAW=$(echo "$MEM_LINE" | sed -E 's/Mem Storage: ([^/]+) \/.*/\1/' | xargs) +CAPACITY_RAW=$(echo "$MEM_LINE" | sed -E 's/.*\/ ([^(]+) \(.*/\1/' | xargs) +PERCENT_RAW=$(echo "$MEM_LINE" | grep -oE '\([0-9.]+%\)' | tr -d '()%') + +# Normalize units ("2.00 GB" -> "2.00GiB"). +normalize_unit() { + echo "$1" | sed -E 's/ ?GB$/GiB/; s/ ?MB$/MiB/; s/ ?B$/B/' | tr -d ' ' +} +CACHED=$(normalize_unit "$CACHED_RAW") +CACHE_CAPACITY=$(normalize_unit "$CAPACITY_RAW") + +# Report the number of keys as fileNum. +FILE_NUM=$(echo "$RAW" | grep -oE 'Keys: [0-9]+' | grep -oE '[0-9]+' || echo "0") + +# Approximate the hit ratio with the success rate of Get requests. +GET_STATS=$(echo "$RAW" | grep -oE 'Get=[0-9.]+/[0-9.]+' || echo "Get=0.00/0.00") +GET_SUCCESS=$(echo "$GET_STATS" | cut -d= -f2 | cut -d/ -f1) +GET_TOTAL=$(echo "$GET_STATS" | cut -d/ -f2) +HIT_RATIO=$(awk -v s="$GET_SUCCESS" -v t="$GET_TOTAL" \ + 'BEGIN{ if (t>0) printf "%.0f", (s/t*100); else print "0" }') + +# Mooncake has no underlying storage, so report the cache capacity as ufsTotal. +UFS_TOTAL="$CACHE_CAPACITY" + +cat <