From cb2822452465669f6ee3287f3b23ecc706ead1cf Mon Sep 17 00:00:00 2001 From: btxu-db Date: Sun, 16 Aug 2026 18:45:21 +0900 Subject: [PATCH 1/3] docs: add a Mooncake CacheRuntime sample for client-less cache systems Mooncake is a distributed KVCache store with no POSIX mount semantics: applications talk to the cache service through its own client library instead of a mount point. Its CacheRuntimeClass therefore declares only master and worker, and omits the client component. Add an end-to-end sample walking through that minimal setup, and document the client-less topology in the generic cache runtime integration guide, including the fact that Fluid still creates a Bound PVC/PV that application pods must not mount. Requires #6157, without which the controller panics when a topology omits the client component. Signed-off-by: btxu-db --- docs/en/TOC.md | 1 + .../dev/generic_cache_runtime_integration.md | 11 + .../cacheruntime/mooncake_cache_runtime.md | 458 ++++++++++++++++++ docs/zh/TOC.md | 1 + .../dev/generic_cache_runtime_integration.md | 11 + .../cacheruntime/mooncake_cache_runtime.md | 458 ++++++++++++++++++ 6 files changed, 940 insertions(+) create mode 100644 docs/en/samples/cacheruntime/mooncake_cache_runtime.md create mode 100644 docs/zh/samples/cacheruntime/mooncake_cache_runtime.md diff --git a/docs/en/TOC.md b/docs/en/TOC.md index 2dc88c733aa..75773aa8511 100644 --- a/docs/en/TOC.md +++ b/docs/en/TOC.md @@ -23,6 +23,7 @@ - [Data Preloading](samples/data_warmup.md) - [CacheRuntime Data Operations](samples/cacheruntime/cacheruntime_data_operations.md) - [Use Curvine as CacheRuntime for Data Caching](samples/cacheruntime/curvine_cache_runtime.md) + - [Deploy Mooncake with CacheRuntime](samples/cacheruntime/mooncake_cache_runtime.md) - [Cache Runtime Manually Scaling](samples/dataset_scaling.md) - [Automatic Cleanup Data Operation](samples/automatic_clean_up_data_operation.md) + Security diff --git a/docs/en/dev/generic_cache_runtime_integration.md b/docs/en/dev/generic_cache_runtime_integration.md index 54f899458ed..3ec92dcbd0a 100644 --- a/docs/en/dev/generic_cache_runtime_integration.md +++ b/docs/en/dev/generic_cache_runtime_integration.md @@ -176,6 +176,17 @@ topology: imagePullPolicy: IfNotPresent ``` +#### Cache Systems Without a Client Component + +The master, worker and client components under `topology` are all optional in the API. You may omit the **client** component if the underlying cache system has both of the following characteristics: + +1. **No POSIX mount semantics**: the cache system itself provides no FUSE-like mount capability. +2. **Applications connect directly**: applications read and write against the master/worker through the cache system's own SDK or RPC client, without relying on a mount point. + +In that case the Master and Worker components start as usual, the Dataset reaches the Bound phase as usual, and cache status is still reported through the ReportSummary script. + +Note that Fluid still creates a PVC/PV for such a Dataset and reports them as Bound, but an application pod that mounts the PVC stays in ContainerCreating forever (reporting `timeout waiting for FUSE mount point`). For this kind of cache system, applications should talk to the cache service directly instead of going through the Dataset → PVC → volumeMounts path. For a complete configuration example, see [Deploy Mooncake with CacheRuntime](../samples/cacheruntime/mooncake_cache_runtime.md). + ### Step 2.5 User Creates Runtime ```yaml diff --git a/docs/en/samples/cacheruntime/mooncake_cache_runtime.md b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md new file mode 100644 index 00000000000..32bb1b574b5 --- /dev/null +++ b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md @@ -0,0 +1,458 @@ +# Example - Deploy Mooncake with CacheRuntime + +[Mooncake](https://github.com/kvcache-ai/Mooncake) is a distributed KVCache store built for LLM inference workloads. Unlike cache systems such as Alluxio or JuiceFS, Mooncake does not provide POSIX mount semantics: applications talk to the cache service directly through its own client library instead of reading and writing files through a mount point. + +Fluid's generic CacheRuntime supports this kind of cache system: the CacheRuntimeClass `topology` declares only the master and worker components, and omits the client component that would otherwise be responsible for mounting. This document walks through that minimal setup. + +For background on the client-less architecture, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## Prerequisites + +Before running this example, follow the [Installation Guide](../../userguide/install.md) to install Fluid, and verify that its components are running: + +```shell +$ kubectl get pod -n fluid-system +NAME READY STATUS RESTARTS AGE +cacheruntime-controller-58c775584-x9q47 1/1 Running 0 10m +csi-nodeplugin-fluid-74bpw 2/2 Running 0 10m +csi-nodeplugin-fluid-cjvc6 2/2 Running 0 10m +csi-nodeplugin-fluid-x4wwp 2/2 Running 0 10m +dataset-controller-58497b968b-q6ssv 1/1 Running 0 10m +fluid-webhook-74db64fdf4-7rq69 1/1 Running 0 10m +fluidapp-controller-7dbdc7696b-86pph 1/1 Running 0 10m +``` + +Typically you should see one pod each for `dataset-controller`, `cacheruntime-controller`, `fluid-webhook` and `fluidapp-controller`, plus one `csi-nodeplugin` pod per node. `cacheruntime-controller` is the one this example requires; it is enabled with `--set runtime.cacheruntime.enabled=true` when installing the Helm chart. + +> Note: this example requires a Fluid build that includes [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157). On earlier versions the controller panics when the client component is omitted from `topology`. + +The Mooncake image used in this example: + +``` +btxu/mooncake:v3 +``` + +An Alibaba Cloud mirror is also available for networks in mainland China: + +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3 +``` + +The image adds two scripts on top of the official Mooncake image for Fluid to invoke: + +| Path | Purpose | +|---|---| +| `/custom-entrypoint.sh` | Component entrypoint; starts the right process based on the role (master/worker) | +| `/reportSummary.sh` | Collects cache usage and emits it as JSON in the format Fluid expects | + +Neither script is specific to this image: you can build an equivalent image yourself on top of the official Mooncake image, as long as it provides the same two entry points. See the conventions in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## Running the Example + +### Create the CacheRuntimeClass + +**Review the CacheRuntimeClass to be created** + +```shell +$ cat<mooncake-cacheruntimeclass.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntimeClass +metadata: + name: mooncake-demo +fileSystemType: mooncakefs +topology: + master: + service: + headless: {} + executionEntries: + reportSummary: + command: + - bash + - -c + - /reportSummary.sh + timeout: 30 + template: + spec: + restartPolicy: Always + containers: + - name: master + image: btxu/mooncake:v3 + command: + - /custom-entrypoint.sh + args: + - master + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50051 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50051 + name: rpc + - containerPort: 8080 + name: metadata + - containerPort: 9003 + name: metrics + worker: + service: + headless: {} + template: + spec: + restartPolicy: Always + containers: + - name: worker + image: btxu/mooncake:v3 + command: + - /custom-entrypoint.sh + args: + - worker + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50052 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50052 + name: data + - containerPort: 9300 + name: http +EOF +``` + +`CacheRuntimeClass` is a CRD defined by Fluid that describes how a class of cache system runs on Kubernetes — the image each component uses, how it starts, which ports it exposes, and how Fluid collects its runtime status. + +A few things worth noting in this example: + +- Only `master` and `worker` are declared under `topology`. All three components (master, worker, client) are optional in the API, and Mooncake does not need a client component. +- Note that `fileSystemType` and `topology` are top-level fields — they do not live under `spec`. +- The master component exposes three ports: `50051` for RPC, `8080` for the metadata service, and `9003` for metrics. Applications connect directly to the first two. +- `reportSummary` points at `/reportSummary.sh` inside the image. Fluid executes it periodically in the component pod and writes the result to the Dataset's `status` field. For the required output format, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +**Create the CacheRuntimeClass** + +```shell +$ kubectl apply -f mooncake-cacheruntimeclass.yaml +cacheruntimeclass.data.fluid.io/mooncake-demo created + +$ kubectl get cacheruntimeclass +NAME AGE +mooncake-demo 0s +``` + +> Tip: if you modify the CacheRuntimeClass afterwards, you must delete and recreate the CacheRuntime for the change to take effect. A CacheRuntime renders the CacheRuntimeClass into workloads at creation time; later edits are not applied retroactively, and deleting the pods alone does not help. + +### Create the Dataset and CacheRuntime + +**Review the resources to be created** + +```shell +$ cat<mooncake-dataset-runtime.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: Dataset +metadata: + name: mooncake-demo + namespace: default +spec: + placement: Shared + accessModes: + - ReadWriteMany + mounts: + - name: mc + mountPoint: "mooncakefs:///" +--- +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntime +metadata: + name: mooncake-demo + namespace: default +spec: + runtimeClassName: mooncake-demo + master: + replicas: 1 + worker: + replicas: 2 + tieredStore: + levels: + - emptyDir: + quota: 1Gi + high: "0.8" + low: "0.5" +EOF +``` + +The `Dataset` describes the dataset itself, while the `CacheRuntime` describes the runtime instance providing cache service for it — here, one master replica and two worker replicas. + +About the `mounts` field: Mooncake does not mount any underlying storage (UFS); data is written directly into the cache by the client, so `mooncakefs:///` here is only a placeholder mount point. Fluid passes the contents of `mounts` through into the CacheRuntime config ConfigMap for the component containers to read, but because this example's CacheRuntimeClass declares no `mountUfs` execution entry, Fluid never performs an actual mount (see the notes on skipping MountUFS in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md)). In the API `mounts` is optional, but if you do declare it, it must contain at least one entry. + +**Create the resources** + +```shell +$ kubectl apply -f mooncake-dataset-runtime.yaml +dataset.data.fluid.io/mooncake-demo created +cacheruntime.data.fluid.io/mooncake-demo created +``` + +**Check the Dataset status** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo NotBound 6s +``` + +As shown above, `PHASE` is `NotBound`, meaning the `Dataset` is not yet bound to a cache runtime. Wait for the components to start: + +```shell +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-demo-master-0 1/1 Running 0 15s 10.244.2.13 fluid-mooncake-worker +mooncake-demo-worker-0 1/1 Running 0 15s 10.244.2.14 fluid-mooncake-worker +mooncake-demo-worker-1 1/1 Running 0 15s 10.244.1.7 fluid-mooncake-worker2 +``` + +**Check the Dataset status again** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo 2.00GiB 0B 2.00GiB 0.0 Bound 11s +``` + +`PHASE` is now `Bound` and the cache service is ready to use. + +> Note: as with other runtimes, Fluid also creates a PV and PVC for this Dataset, and they show as Bound. But since there is no client component, there is no FUSE mount point, so **application pods must not mount this PVC**. See [FAQ](#faq) at the end of this document. + +## Accessing the Cache + +Mooncake applications read and write data by connecting directly to the master service through its Python client, not through a mount point. + +**Review the application to be created** + +```shell +$ cat<mooncake-client.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-1 +spec: + nodeName: fluid-mooncake-worker2 + containers: + - name: client + image: btxu/mooncake:v3 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF +``` + +Note that this pod's `spec` has no `volumes` or `volumeMounts` — this is the main difference between this example and the other runtime examples. + +A few notes: + +- This example reuses the Mooncake image as the client environment since it already ships the Python client library. `command` is overridden with `sleep infinity` so the image's default entrypoint does not start it as a master or worker. +- `POD_IP` is injected via the downward API. The client's `local_hostname` must be the pod's own IP so that other nodes can connect back to fetch data. +- `nodeName` pins the pod to a specific node so that the cross-node read later is deterministic: the writer stays on one node and the reader on another. Replace `fluid-mooncake-worker2` with a real node name from your cluster (`kubectl get nodes`), and make sure it differs from the node used by client-2 below. + +**Start the application and write data** + +```shell +$ kubectl apply -f mooncake-client.yaml +$ kubectl exec -it mooncake-client-1 -- python3 +``` + +```python +import os, hashlib +from mooncake.store import MooncakeDistributedStore + +MASTER = "mooncake-demo-master-0.svc-mooncake-demo-master" + +store = MooncakeDistributedStore() +store.setup( + local_hostname=os.environ["POD_IP"], + metadata_server=f"http://{MASTER}:8080/metadata", + master_server_addr=f"{MASTER}:50051", + global_segment_size=0, + local_buffer_size=128 * 1024 * 1024, + protocol="tcp", + rdma_devices="", +) + +payload = os.urandom(4 * 1024 * 1024) +store.put("demo_key", payload) +got = store.get("demo_key") +print("put md5:", hashlib.md5(payload).hexdigest()) +print("get md5:", hashlib.md5(got).hexdigest()) +print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) +``` + +> The connection address is the master pod's DNS name `mooncake-demo-master-0.svc-mooncake-demo-master`, not the Service name. The CacheRuntimeClass declares a headless Service, which does not expose ports itself, so you must reach the pod through its DNS record. + +Output of `setup()` (key lines only): + +``` +I0815 05:30:15.076617 transfer_metadata_plugin.cpp:1293] Found active interface eth0 with IP 10.244.1.8 +I0815 05:30:15.077410 client_service.cpp:747] Transfer engine auto discovery is disabled for protocol: tcp +I0815 05:30:15.078089 real_client.cpp:734] Successfully created client on port 12699 after 1 attempt(s) +I0815 05:30:15.079909 real_client.cpp:767] Registering local memory: 134217728 bytes +I0815 05:30:15.080171 real_client.cpp:932] Global segment size is 0, skip mounting segment +0 +``` + +> With `global_segment_size` set to 0, the client log shows `Global segment size is 0, skip mounting segment`, meaning the client no longer allocates a local memory segment and all data is held by the Fluid-managed workers. +> +> The `http=404 body: metadata not found` message during startup is expected: the client is registering itself with the metadata service for the first time, so the corresponding key does not exist yet. + +Read/write result: + +``` +put md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +get md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +match: True +``` + +> Note: Mooncake's `put` silently skips a key that **already exists** — it does not overwrite the old data, and it still returns `0`, so the return value gives nothing away. If you run the snippet above a second time (or also run a write in client-2 later on), the second `put` has no effect and `get` returns the content written the first time, producing `match: False`. This is not a cache failure. Before re-running this example, use a different key name, or delete and recreate the CacheRuntime so the workers' cache is cleared. + +**Reading across pods and nodes** + +First exit the Python session in the first client so the writer process terminates: + +```python +>>> exit() +``` + +Then create a second client pod, with `nodeName` pointing at a **different** node (client-1 is on `fluid-mooncake-worker2`, so this one uses `fluid-mooncake-worker`): + +```shell +$ cat<mooncake-client-2.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-2 +spec: + nodeName: fluid-mooncake-worker + containers: + - name: client + image: btxu/mooncake:v3 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF + +$ kubectl apply -f mooncake-client-2.yaml +$ kubectl get pod mooncake-client-1 mooncake-client-2 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-mooncake-worker2 +mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker +``` + +In the second pod, only read and verify (the `setup` parameters are the same as above): + +```python +got = store.get("demo_key") +print("len:", len(got)) +print("md5:", hashlib.md5(got).hexdigest()) +``` + +Output: + +``` +len: 4194304 +md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +``` + +The md5 matches the writer exactly. Note that by this point the client process that wrote the data has already exited, and the reader is on a different node — showing that the cached data is held by the Fluid-managed workers, depending neither on the client process that wrote it nor on the node it ran on. + +## Inspecting Cache Status + +```shell +$ kubectl get dataset mooncake-demo -o yaml +``` + +> Note: Fluid runs the ReportSummary script periodically, so the Dataset `status` does not update the moment a write completes — in practice it takes about a minute to refresh. Until then you will still see `cached: 0B` and `fileNum: "0"`, which is expected; just check again shortly. To see the cache system's live state directly, run the script inside the master pod: +> +> ```shell +> $ kubectl exec mooncake-demo-master-0 -- bash -c /reportSummary.sh +> ``` + +```yaml + cacheStates: + cacheCapacity: 2.00GiB + cacheHitRatio: "0" + cached: 4.00MiB + cachedPercentage: "0.2" + fileNum: "1" + ufsTotal: 2.00GiB + conditions: + - lastTransitionTime: "2026-08-15T05:28:03Z" + lastUpdateTime: "2026-08-15T05:28:03Z" + message: The ddc runtime is ready. + reason: DatasetReady + status: "True" + type: Ready +``` + +`cached: 4.00MiB` matches the 4 MiB written above, and `fileNum: 1` corresponds to the single key. `cacheCapacity: 2.00GiB` comes from the 1Gi `tieredStore` quota on each of the two workers. + +The remaining two fields need to be read in the context of having no UFS: + +- `ufsTotal` normally reports the total data size in the underlying storage. Mooncake has no UFS, so the example image's `reportSummary.sh` fills the field with the total cache capacity instead (`UFS_TOTAL="$CACHE_CAPACITY"` in the script), which is why it equals `cacheCapacity`. This is a choice made by the script, not an anomaly. +- `cacheHitRatio` is approximated by the script from the master's `Get` request success rate. Note that `Requests (Success/Total per sec)` in the master's `/metrics/summary` is a **per-second instantaneous rate**, not a cumulative counter, so if no Get is in flight when the script samples, it reads `0.00/0.00` — which is why this field usually shows `0`. It reflects request success rate at the sampling instant, not a strict cache hit ratio. + +All of these values are collected and reported by the ReportSummary script configured in the CacheRuntimeClass; Fluid only passes them through. For the meaning of each field and the required output format, see the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). + +## FAQ + +### An application pod is stuck in ContainerCreating + +**Symptom**: an application pod that mounts this Dataset's PVC never starts, and its events show `FailedMount`. + +```shell +$ kubectl describe pod +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 60s default-scheduler Successfully assigned default/ to fluid-mooncake-worker + Warning FailedMount 29s kubelet MountVolume.SetUp failed for volume "default-mooncake-demo" : rpc error: code = Internal desc = timeout waiting for FUSE mount point to be ready +``` + +**Cause**: this example's CacheRuntimeClass declares no client component, so there is no FUSE mount point. The CSI plugin waits for a mount point that never appears until it times out, which produces the `timeout waiting for FUSE mount point to be ready` above. Fluid still creates the PVC/PV and reports them as Bound, but application pods cannot consume them through `volumeMounts`. + +**Resolution**: remove the PVC mount from the application pod, connect to the cache service directly as shown above, and delete the stuck pod. + +```shell +$ kubectl delete pod +``` + +## Cleanup + +```shell +$ kubectl delete -f mooncake-client-2.yaml +$ kubectl delete -f mooncake-client.yaml +$ kubectl delete -f mooncake-dataset-runtime.yaml +$ kubectl delete -f mooncake-cacheruntimeclass.yaml +``` diff --git a/docs/zh/TOC.md b/docs/zh/TOC.md index d00a1825043..eab87e5585c 100644 --- a/docs/zh/TOC.md +++ b/docs/zh/TOC.md @@ -28,6 +28,7 @@ - [数据预加载](samples/data_warmup.md) - [CacheRuntime 数据操作](samples/cacheruntime/cacheruntime_data_operations.md) - [使用 Curvine 作为 CacheRuntime 进行数据缓存](samples/cacheruntime/curvine_cache_runtime.md) + - [使用 CacheRuntime 部署 Mooncake](samples/cacheruntime/mooncake_cache_runtime.md) - [Cache Runtime手动扩缩容](samples/dataset_scaling.md) - [数据操作自动清理](samples/automatic_clean_up_data_operation.md) + 安全 diff --git a/docs/zh/dev/generic_cache_runtime_integration.md b/docs/zh/dev/generic_cache_runtime_integration.md index 14d666e890c..6c63f121889 100644 --- a/docs/zh/dev/generic_cache_runtime_integration.md +++ b/docs/zh/dev/generic_cache_runtime_integration.md @@ -175,6 +175,17 @@ topology: - custom-endpoint.sh imagePullPolicy: IfNotPresent ``` +#### 无 Client 组件的场景 + +topology 中的 master、worker、client 三个组件在 API 上均为可选。如果底层缓存系统具备以下特征，可以**不声明 client**： + +1. **无 POSIX 挂载语义**：缓存系统本身不提供 FUSE 之类的挂载能力 +2. **应用直连访问**：应用通过缓存系统自带的 SDK 或 RPC 客户端直接读写 master/worker，不依赖挂载点 + +此时 Master 与 Worker 组件正常拉起，Dataset 正常进入 Bound 状态，缓存状态也会通过 ReportSummary 脚本正常上报。 + +需要注意的是，Fluid 仍会为该 Dataset 创建 PVC/PV 且状态为 Bound，但业务 Pod 挂载该 PVC 会一直停留在 ContainerCreating（报 timeout waiting for FUSE mount point）。这类缓存系统应由应用直连缓存服务读写，不要走 Dataset → PVC → volumeMounts 这条路径。完整配置示例参见 [使用 CacheRuntime 部署 Mooncake](../samples/cacheruntime/mooncake_cache_runtime.md)。 + ### 步骤2.5 用户创建Runtime diff --git a/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md new file mode 100644 index 00000000000..09902fc51be --- /dev/null +++ b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md @@ -0,0 +1,458 @@ +# 示例 - 使用 CacheRuntime 部署 Mooncake + +[Mooncake](https://github.com/kvcache-ai/Mooncake) 是一个面向大模型推理场景的分布式 KVCache 存储系统。与 Alluxio、JuiceFS 等缓存系统不同，Mooncake 不提供 POSIX 挂载语义，应用通过其自带的客户端直接与缓存服务通信，而不是通过挂载点读写文件。 + +Fluid 的通用 CacheRuntime 支持这类缓存系统：在 CacheRuntimeClass 的 `topology` 中只声明 master 和 worker 组件，不声明负责挂载的 client 组件。本文档演示这一最小实现。 + +关于无 client 架构的说明，参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 前提条件 + +在运行该示例之前，请参考[安装文档](../../userguide/install.md)完成安装，并检查 Fluid 各组件正常运行： + +```shell +$ kubectl get pod -n fluid-system +NAME READY STATUS RESTARTS AGE +cacheruntime-controller-58c775584-x9q47 1/1 Running 0 10m +csi-nodeplugin-fluid-74bpw 2/2 Running 0 10m +csi-nodeplugin-fluid-cjvc6 2/2 Running 0 10m +csi-nodeplugin-fluid-x4wwp 2/2 Running 0 10m +dataset-controller-58497b968b-q6ssv 1/1 Running 0 10m +fluid-webhook-74db64fdf4-7rq69 1/1 Running 0 10m +fluidapp-controller-7dbdc7696b-86pph 1/1 Running 0 10m +``` + +通常来说，你会看到 `dataset-controller`、`cacheruntime-controller`、`fluid-webhook`、`fluidapp-controller` 各一个 Pod，以及每个节点一个的 `csi-nodeplugin` Pod 正在运行。其中 `cacheruntime-controller` 是本示例必需的，它由 Helm 安装时的 `--set runtime.cacheruntime.enabled=true` 开启。 + +> 注意：本示例需要包含 [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157) 的 Fluid 版本。较早版本在 `topology` 中省略 client 组件时 controller 会发生 panic。 + +本示例使用的 Mooncake 镜像： + +``` +btxu/mooncake:v3 +``` + +国内网络环境下也可使用阿里云镜像： + +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3 +``` + +该镜像在官方 Mooncake 镜像基础上加入了两个脚本，供 Fluid 调用： + +| 路径 | 作用 | +|---|---| +| `/custom-entrypoint.sh` | 组件启动入口，按角色（master/worker）启动对应进程 | +| `/reportSummary.sh` | 采集缓存用量并按 Fluid 要求的 JSON 格式输出 | + +这两个脚本并非该镜像独有：你也可以基于官方 Mooncake 镜像自行构建等价镜像，只要它提供同样的两个入口即可，具体约定参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 运行示例 + +### 创建 CacheRuntimeClass + +**查看待创建的 CacheRuntimeClass 资源对象** + +```shell +$ cat<mooncake-cacheruntimeclass.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntimeClass +metadata: + name: mooncake-demo +fileSystemType: mooncakefs +topology: + master: + service: + headless: {} + executionEntries: + reportSummary: + command: + - bash + - -c + - /reportSummary.sh + timeout: 30 + template: + spec: + restartPolicy: Always + containers: + - name: master + image: btxu/mooncake:v3 + command: + - /custom-entrypoint.sh + args: + - master + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50051 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50051 + name: rpc + - containerPort: 8080 + name: metadata + - containerPort: 9003 + name: metrics + worker: + service: + headless: {} + template: + spec: + restartPolicy: Always + containers: + - name: worker + image: btxu/mooncake:v3 + command: + - /custom-entrypoint.sh + args: + - worker + - start + imagePullPolicy: IfNotPresent + readinessProbe: + tcpSocket: + port: 50052 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 12 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + ports: + - containerPort: 50052 + name: data + - containerPort: 9300 + name: http +EOF +``` + +`CacheRuntimeClass` 是 Fluid 定义的 CRD，用于描述一类缓存系统如何在 Kubernetes 上运行——包括各组件使用的镜像、启动方式、端口，以及 Fluid 如何采集其运行状态。 + +本示例中需要留意的几点： + +- `topology` 下只声明了 `master` 和 `worker`。master、worker、client 三个组件在 API 中均为可选字段，Mooncake 无需 client 组件。 +- 注意 `fileSystemType` 和 `topology` 是顶层字段，不在 `spec` 之下。 +- master 组件暴露三个端口：`50051` 用于 RPC，`8080` 用于元数据服务，`9003` 用于 metrics。应用将直接连接前两个。 +- `reportSummary` 指向镜像中的 `/reportSummary.sh`，Fluid 会周期性地在组件 Pod 中执行它，并将结果更新到 Dataset 的 `status` 字段。其输出格式要求参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +**创建 CacheRuntimeClass 资源对象** + +```shell +$ kubectl apply -f mooncake-cacheruntimeclass.yaml +cacheruntimeclass.data.fluid.io/mooncake-demo created + +$ kubectl get cacheruntimeclass +NAME AGE +mooncake-demo 0s +``` + +> 提示：如果之后修改了 CacheRuntimeClass，需要删除并重建 CacheRuntime 才会生效。CacheRuntime 在创建时就已将当时的 CacheRuntimeClass 渲染为工作负载，后续修改不会回溯更新，仅删除 Pod 也无效。 + +### 创建 Dataset 与 CacheRuntime + +**查看待创建的资源对象** + +```shell +$ cat<mooncake-dataset-runtime.yaml +apiVersion: data.fluid.io/v1alpha1 +kind: Dataset +metadata: + name: mooncake-demo + namespace: default +spec: + placement: Shared + accessModes: + - ReadWriteMany + mounts: + - name: mc + mountPoint: "mooncakefs:///" +--- +apiVersion: data.fluid.io/v1alpha1 +kind: CacheRuntime +metadata: + name: mooncake-demo + namespace: default +spec: + runtimeClassName: mooncake-demo + master: + replicas: 1 + worker: + replicas: 2 + tieredStore: + levels: + - emptyDir: + quota: 1Gi + high: "0.8" + low: "0.5" +EOF +``` + +`Dataset` 描述数据集本身，`CacheRuntime` 描述为该数据集提供缓存服务的运行时实例——本示例中即一个 master 副本和两个 worker 副本。 + +关于 `mounts` 字段：Mooncake 不挂载任何底层存储（UFS），数据由客户端直接写入缓存，因此这里的 `mooncakefs:///` 只是一个占位挂载点。Fluid 会把 `mounts` 的内容透传到 CacheRuntime 的配置 ConfigMap 中，供组件容器读取；但由于本示例的 CacheRuntimeClass 未声明 `mountUfs` 执行入口，Fluid 不会真正执行任何挂载动作（详见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)中关于"可以不设置 MountUFS"的说明）。API 上 `mounts` 是可选字段，但一旦声明就至少需要一项。 + +**创建资源对象** + +```shell +$ kubectl apply -f mooncake-dataset-runtime.yaml +dataset.data.fluid.io/mooncake-demo created +cacheruntime.data.fluid.io/mooncake-demo created +``` + +**查看 Dataset 资源对象状态** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo NotBound 6s +``` + +如上所示，`PHASE` 属性值为 `NotBound`，这意味着该 `Dataset` 资源对象目前还未与缓存运行时绑定。等待各组件启动： + +```shell +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-demo-master-0 1/1 Running 0 15s 10.244.2.13 fluid-mooncake-worker +mooncake-demo-worker-0 1/1 Running 0 15s 10.244.2.14 fluid-mooncake-worker +mooncake-demo-worker-1 1/1 Running 0 15s 10.244.1.7 fluid-mooncake-worker2 +``` + +**再次查看 Dataset 资源对象状态** + +```shell +$ kubectl get dataset +NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE AGE +mooncake-demo 2.00GiB 0B 2.00GiB 0.0 Bound 11s +``` + +此时 `PHASE` 已变为 `Bound`，缓存服务可以使用了。 + +> 注意：与其他 Runtime 一样，Fluid 也会为该 Dataset 创建 PV 和 PVC，且状态显示为 Bound。但由于没有 client 组件，不存在 FUSE 挂载点，**业务 Pod 不应挂载该 PVC**。详见文末[常见问题](#常见问题)。 + +## 访问缓存 + +Mooncake 的应用通过其 Python 客户端直连 master 服务读写数据，不经过挂载点。 + +**查看待创建的应用** + +```shell +$ cat<mooncake-client.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-1 +spec: + nodeName: fluid-mooncake-worker2 + containers: + - name: client + image: btxu/mooncake:v3 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF +``` + +注意该 Pod 的 `spec` 中没有 `volumes` 和 `volumeMounts`——这是本示例与其他 Runtime 示例最主要的区别。 + +几点说明： + +- 本示例直接复用 Mooncake 镜像作为客户端环境，它已包含 Python 客户端库；`command` 覆盖为 `sleep infinity`，避免镜像的默认入口把它启动成 master 或 worker。 +- `POD_IP` 通过 downward API 注入，客户端的 `local_hostname` 必须是 Pod 自身 IP，其他节点才能回连取数。 +- `nodeName` 显式指定节点，是为了让后面的跨节点读取有确定的效果：写入方固定在一个节点，读取方固定在另一个节点。请把 `fluid-mooncake-worker2` 换成你集群中的实际节点名（`kubectl get nodes`），并确保与后面 client-2 使用的节点不同。 + +**启动应用并写入数据** + +```shell +$ kubectl apply -f mooncake-client.yaml +$ kubectl exec -it mooncake-client-1 -- python3 +``` + +```python +import os, hashlib +from mooncake.store import MooncakeDistributedStore + +MASTER = "mooncake-demo-master-0.svc-mooncake-demo-master" + +store = MooncakeDistributedStore() +store.setup( + local_hostname=os.environ["POD_IP"], + metadata_server=f"http://{MASTER}:8080/metadata", + master_server_addr=f"{MASTER}:50051", + global_segment_size=0, + local_buffer_size=128 * 1024 * 1024, + protocol="tcp", + rdma_devices="", +) + +payload = os.urandom(4 * 1024 * 1024) +store.put("demo_key", payload) +got = store.get("demo_key") +print("put md5:", hashlib.md5(payload).hexdigest()) +print("get md5:", hashlib.md5(got).hexdigest()) +print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) +``` + +> 连接地址使用的是 master Pod 的 DNS 名 `mooncake-demo-master-0.svc-mooncake-demo-master`，而不是 Service 名。CacheRuntimeClass 中声明的是 headless Service，其本身不暴露端口，需要通过 Pod 的 DNS 记录访问。 + +`setup()` 的输出（截取关键部分）： + +``` +I0815 05:30:15.076617 transfer_metadata_plugin.cpp:1293] Found active interface eth0 with IP 10.244.1.8 +I0815 05:30:15.077410 client_service.cpp:747] Transfer engine auto discovery is disabled for protocol: tcp +I0815 05:30:15.078089 real_client.cpp:734] Successfully created client on port 12699 after 1 attempt(s) +I0815 05:30:15.079909 real_client.cpp:767] Registering local memory: 134217728 bytes +I0815 05:30:15.080171 real_client.cpp:932] Global segment size is 0, skip mounting segment +0 +``` + +> `global_segment_size` 设为 0 时，客户端日志中会出现 `Global segment size is 0, skip mounting segment`，表示客户端不再分配本地内存段，数据全部由 Fluid 管理的 worker 承载。 +> +> 启动过程中出现的 `http=404 body: metadata not found` 属于正常现象：客户端首次向元数据服务注册自身时，对应的 key 尚不存在。 + +读写结果： + +``` +put md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +get md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +match: True +``` + +> 注意：Mooncake 的 `put` 对**已存在**的 key 是静默跳过的——不覆盖旧数据，且仍然返回 `0`，从返回值上看不出区别。因此如果你重复执行上面这段代码（或在后面的 client-2 里也执行了写入），第二次的 `put` 不会生效，`get` 读回的仍是第一次写入的内容，于是出现 `match: False`。这不是缓存故障。重跑本示例前，请换一个 key 名，或删除并重建 CacheRuntime 让 worker 的缓存清空。 + +**跨 Pod 跨节点读取** + +先退出第一个客户端的 Python 交互环境，让写入方进程结束： + +```python +>>> exit() +``` + +再创建第二个客户端 Pod，`nodeName` 指向**另一个**节点（client-1 在 `fluid-mooncake-worker2`，这里用 `fluid-mooncake-worker`）： + +```shell +$ cat<mooncake-client-2.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mooncake-client-2 +spec: + nodeName: fluid-mooncake-worker + containers: + - name: client + image: btxu/mooncake:v3 + imagePullPolicy: IfNotPresent + command: ["sleep", "infinity"] + env: + - name: POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP +EOF + +$ kubectl apply -f mooncake-client-2.yaml +$ kubectl get pod mooncake-client-1 mooncake-client-2 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-mooncake-worker2 +mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker +``` + +在第二个 Pod 中只做读取和校验（`setup` 参数与上文相同）： + +```python +got = store.get("demo_key") +print("len:", len(got)) +print("md5:", hashlib.md5(got).hexdigest()) +``` + +输出： + +``` +len: 4194304 +md5: e114c0f1fb62bb7b1df45dbb1ab1d201 +``` + +md5 与写入方完全一致。需要强调的是，此时写入数据的那个客户端进程已经退出，且读取方位于另一个节点上——这说明缓存数据由 Fluid 管理的 worker 承载，既不依赖写入它的客户端进程，也不依赖所在节点。 + +## 查看缓存状态 + +```shell +$ kubectl get dataset mooncake-demo -o yaml +``` + +> 注意：Fluid 是周期性执行 ReportSummary 脚本的，写入完成后 Dataset 的 `status` 不会立刻更新。实测约需 1 分钟左右才会刷新，在此之前看到的仍是 `cached: 0B`、`fileNum: "0"`，属于正常现象，稍等再查即可。若想确认缓存系统侧的即时状态，可以直接在 master Pod 中执行该脚本： +> +> ```shell +> $ kubectl exec mooncake-demo-master-0 -- bash -c /reportSummary.sh +> ``` + +```yaml + cacheStates: + cacheCapacity: 2.00GiB + cacheHitRatio: "0" + cached: 4.00MiB + cachedPercentage: "0.2" + fileNum: "1" + ufsTotal: 2.00GiB + conditions: + - lastTransitionTime: "2026-08-15T05:28:03Z" + lastUpdateTime: "2026-08-15T05:28:03Z" + message: The ddc runtime is ready. + reason: DatasetReady + status: "True" + type: Ready +``` + +`cached: 4.00MiB` 与上文写入的 4 MiB 数据一致，`fileNum: 1` 对应写入的一个 key。`cacheCapacity: 2.00GiB` 则来自两个 worker 各 1Gi 的 `tieredStore` 配额。 + +其余两个字段需要结合无 UFS 的场景理解： + +- `ufsTotal` 在其他 Runtime 中表示底层存储的数据总量。Mooncake 没有 UFS，示例镜像的 `reportSummary.sh` 直接用缓存总容量填充该字段（脚本中即 `UFS_TOTAL="$CACHE_CAPACITY"`），因此它与 `cacheCapacity` 相等。这是脚本的上报选择，不是异常。 +- `cacheHitRatio` 由脚本从 master 的 `Get` 请求成功率近似估算。注意 master `/metrics/summary` 中的 `Requests (Success/Total per sec)` 是**每秒瞬时速率**而非累计计数，采样时若没有正在进行的 Get 请求，读到的就是 `0.00/0.00`，因此该字段通常显示为 `0`。它反映的是采样瞬间的请求成功率，并不是严格意义上的缓存命中率。 + +这些数据均由 CacheRuntimeClass 中配置的 ReportSummary 脚本采集上报，Fluid 只做透传，各字段的语义和输出格式要求参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 + +## 常见问题 + +### 业务 Pod 一直处于 ContainerCreating + +**现象**：业务 Pod 挂载了该 Dataset 对应的 PVC 后一直无法启动，事件中出现 `FailedMount`。 + +```shell +$ kubectl describe pod +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 60s default-scheduler Successfully assigned default/ to fluid-mooncake-worker + Warning FailedMount 29s kubelet MountVolume.SetUp failed for volume "default-mooncake-demo" : rpc error: code = Internal desc = timeout waiting for FUSE mount point to be ready +``` + +**原因**：本示例的 CacheRuntimeClass 未声明 client 组件，因此不存在 FUSE 挂载点。CSI 插件会一直等待挂载点就绪直到超时，于是报出上面的 `timeout waiting for FUSE mount point to be ready`。虽然 Fluid 仍会创建 PVC/PV 且状态为 Bound，但业务 Pod 无法通过 `volumeMounts` 使用它。 + +**处理**：从业务 Pod 中移除该 PVC 的挂载，改为按上文方式直连缓存服务，然后删除卡住的 Pod。 + +```shell +$ kubectl delete pod +``` + +## 环境清理 + +```shell +$ kubectl delete -f mooncake-client-2.yaml +$ kubectl delete -f mooncake-client.yaml +$ kubectl delete -f mooncake-dataset-runtime.yaml +$ kubectl delete -f mooncake-cacheruntimeclass.yaml +``` \ No newline at end of file From ae3504cf5c713b66d5006f6d115795573f6bce1a Mon Sep 17 00:00:00 2001 From: btxu-db Date: Thu, 27 Aug 2026 22:17:57 +0900 Subject: [PATCH 2/3] docs: address review feedback on the Mooncake sample Check the demo image's build context into samples/mooncake/docker so the sample can be rebuilt from the repository instead of depending on an opaque image in a personal registry. Pin both the Docker Hub image and its Alibaba Cloud mirror to an immutable digest, note that the two registries are personal accounts with no availability guarantee, and recommend building and substituting your own image. Correct the note on addressing the master: Fluid creates the component Service as headless with no ports, but that only affects SRV records, so the Service name resolves to the backing pod IPs just as the per-pod name does. The per-pod name is preferred only because it stays pinned to a single replica. Show the second client pod opening its own Python session and repeating the MooncakeDistributedStore initialization, which the reader previously had to infer; following the sample literally hit a NameError on store. Also add the trailing newline missing from the Chinese sample. Signed-off-by: btxu-db --- .../cacheruntime/mooncake_cache_runtime.md | 53 +++++++++----- .../cacheruntime/mooncake_cache_runtime.md | 55 +++++++++----- samples/mooncake/docker/Dockerfile | 27 +++++++ samples/mooncake/docker/custom-entrypoint.sh | 73 +++++++++++++++++++ samples/mooncake/docker/reportSummary.sh | 50 +++++++++++++ 5 files changed, 223 insertions(+), 35 deletions(-) create mode 100644 samples/mooncake/docker/Dockerfile create mode 100755 samples/mooncake/docker/custom-entrypoint.sh create mode 100755 samples/mooncake/docker/reportSummary.sh diff --git a/docs/en/samples/cacheruntime/mooncake_cache_runtime.md b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md index 32bb1b574b5..f93674f0126 100644 --- a/docs/en/samples/cacheruntime/mooncake_cache_runtime.md +++ b/docs/en/samples/cacheruntime/mooncake_cache_runtime.md @@ -26,26 +26,39 @@ Typically you should see one pod each for `dataset-controller`, `cacheruntime-co > Note: this example requires a Fluid build that includes [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157). On earlier versions the controller panics when the client component is omitted from `topology`. -The Mooncake image used in this example: +### The demo image -``` -btxu/mooncake:v3 +Fluid does not publish a Mooncake image. This example uses a demo image built from the Mooncake Python distribution plus two small scripts that Fluid invokes: + +| Path | Purpose | +|---|---| +| `/custom-entrypoint.sh` | Component entrypoint; starts the right process based on the role (master/worker) | +| `/reportSummary.sh` | Collects cache usage and emits it as JSON in the format Fluid expects | + +The build context for that image lives in this repository under [`samples/mooncake/docker`](../../../../samples/mooncake/docker), so you can build an equivalent image yourself: + +```shell +$ docker build -t /mooncake:v3 samples/mooncake/docker +$ docker push /mooncake:v3 ``` -An Alibaba Cloud mirror is also available for networks in mainland China: +Note that `apt-get` and `pip` resolve to whatever versions are current at build time, so a fresh build produces a functionally equivalent image, not a bit-for-bit reproduction of the digest below. + +**Building your own image and substituting it in the manifests below is the recommended path.** For convenience, a prebuilt copy is also available, pinned to an immutable digest so this example keeps working even if the tag is moved: ``` -crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3 +btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 ``` -The image adds two scripts on top of the official Mooncake image for Fluid to invoke: +An Alibaba Cloud mirror is available for networks in mainland China. It is a copy of the same manifest, so it carries the identical digest: -| Path | Purpose | -|---|---| -| `/custom-entrypoint.sh` | Component entrypoint; starts the right process based on the role (master/worker) | -| `/reportSummary.sh` | Collects cache usage and emits it as JSON in the format Fluid expects | +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` -Neither script is specific to this image: you can build an equivalent image yourself on top of the official Mooncake image, as long as it provides the same two entry points. See the conventions in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). +> Note: both registries are personal accounts belonging to the author of this example, not project-controlled infrastructure, and they carry no availability guarantee. Treat them as a convenience for trying the example out, and build from `samples/mooncake/docker` for anything beyond that. + +Neither script is specific to this image: you can build an equivalent image on top of any Mooncake distribution, as long as it provides the same two entry points. See the conventions in the [Generic Cache System Integration Guide](../../dev/generic_cache_runtime_integration.md). ## Running the Example @@ -76,7 +89,7 @@ topology: restartPolicy: Always containers: - name: master - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 command: - /custom-entrypoint.sh args: @@ -109,7 +122,7 @@ topology: restartPolicy: Always containers: - name: worker - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 command: - /custom-entrypoint.sh args: @@ -254,7 +267,7 @@ spec: nodeName: fluid-mooncake-worker2 containers: - name: client - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 imagePullPolicy: IfNotPresent command: ["sleep", "infinity"] env: @@ -305,7 +318,7 @@ print("get md5:", hashlib.md5(got).hexdigest()) print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) ``` -> The connection address is the master pod's DNS name `mooncake-demo-master-0.svc-mooncake-demo-master`, not the Service name. The CacheRuntimeClass declares a headless Service, which does not expose ports itself, so you must reach the pod through its DNS record. +> The example connects through the master pod's stable DNS name `mooncake-demo-master-0.svc-mooncake-demo-master`, which pins the client to one specific replica. The Service name `svc-mooncake-demo-master` works just as well here: Fluid creates the component Service as headless (`clusterIP: None`) and declares no `ports` on it, but that only affects SRV records — the A record still resolves straight to the backing pod IPs, and the client connects to container ports `8080` and `50051` directly. The per-pod name is the safer habit because it stays pinned to a single replica if the master is ever scaled out. Output of `setup()` (key lines only): @@ -352,7 +365,7 @@ spec: nodeName: fluid-mooncake-worker containers: - name: client - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 imagePullPolicy: IfNotPresent command: ["sleep", "infinity"] env: @@ -369,7 +382,13 @@ mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-moo mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker ``` -In the second pod, only read and verify (the `setup` parameters are the same as above): +The second pod runs a brand-new Python process, so open a session in it and repeat the imports and the full `MooncakeDistributedStore` initialization shown above — the `setup` parameters are identical, and `POD_IP` again resolves to this pod's own IP: + +```shell +$ kubectl exec -it mooncake-client-2 -- python3 +``` + +Once `store` and `hashlib` are initialized, read and verify: ```python got = store.get("demo_key") diff --git a/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md index 09902fc51be..326ec1803ea 100644 --- a/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md +++ b/docs/zh/samples/cacheruntime/mooncake_cache_runtime.md @@ -26,26 +26,39 @@ fluidapp-controller-7dbdc7696b-86pph 1/1 Running 0 10m > 注意：本示例需要包含 [#6157](https://github.com/fluid-cloudnative/fluid/pull/6157) 的 Fluid 版本。较早版本在 `topology` 中省略 client 组件时 controller 会发生 panic。 -本示例使用的 Mooncake 镜像： +### 示例镜像 -``` -btxu/mooncake:v3 +Fluid 并不发布 Mooncake 镜像。本示例使用的是一个演示镜像，它在 Mooncake 的 Python 发行版之上加入了两个供 Fluid 调用的脚本： + +| 路径 | 作用 | +|---|---| +| `/custom-entrypoint.sh` | 组件启动入口，按角色（master/worker）启动对应进程 | +| `/reportSummary.sh` | 采集缓存用量并按 Fluid 要求的 JSON 格式输出 | + +该镜像的构建上下文已随本仓库提供，位于 [`samples/mooncake/docker`](../../../../samples/mooncake/docker)，你可以据此构建等价镜像： + +```shell +$ docker build -t /mooncake:v3 samples/mooncake/docker +$ docker push /mooncake:v3 ``` -国内网络环境下也可使用阿里云镜像： +注意 `apt-get` 和 `pip` 在构建时拉取的都是当时的最新版本，因此重新构建得到的是功能等价的镜像，并不能字节级复现下面的摘要。 + +**推荐的做法是自行构建镜像，并替换下文各处清单中的镜像地址。** 为方便试用，也提供了一份预构建镜像，并固定到不可变的镜像摘要，这样即使标签被覆盖，本示例仍然可用： ``` -crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3 +btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 ``` -该镜像在官方 Mooncake 镜像基础上加入了两个脚本，供 Fluid 调用： +国内网络环境下也可使用阿里云镜像。它是同一 manifest 的副本，摘要与上面完全相同： -| 路径 | 作用 | -|---|---| -| `/custom-entrypoint.sh` | 组件启动入口，按角色（master/worker）启动对应进程 | -| `/reportSummary.sh` | 采集缓存用量并按 Fluid 要求的 JSON 格式输出 | +``` +crpi-4hkqof7tc9brc6d5.cn-hongkong.personal.cr.aliyuncs.com/mooncake1314/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 +``` + +> 注意：以上两个仓库均为本示例作者的个人账号，并非项目管控的基础设施，不提供任何可用性保证。它们仅用于快速试用本示例；正式使用请基于 `samples/mooncake/docker` 自行构建。 -这两个脚本并非该镜像独有：你也可以基于官方 Mooncake 镜像自行构建等价镜像，只要它提供同样的两个入口即可，具体约定参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 +这两个脚本并非该镜像独有：你也可以基于任意 Mooncake 发行版自行构建等价镜像，只要它提供同样的两个入口即可，具体约定参见[通用缓存系统接入指南](../../dev/generic_cache_runtime_integration.md)。 ## 运行示例 @@ -76,7 +89,7 @@ topology: restartPolicy: Always containers: - name: master - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 command: - /custom-entrypoint.sh args: @@ -109,7 +122,7 @@ topology: restartPolicy: Always containers: - name: worker - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 command: - /custom-entrypoint.sh args: @@ -254,7 +267,7 @@ spec: nodeName: fluid-mooncake-worker2 containers: - name: client - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 imagePullPolicy: IfNotPresent command: ["sleep", "infinity"] env: @@ -305,7 +318,7 @@ print("get md5:", hashlib.md5(got).hexdigest()) print("match:", hashlib.md5(payload).hexdigest() == hashlib.md5(got).hexdigest()) ``` -> 连接地址使用的是 master Pod 的 DNS 名 `mooncake-demo-master-0.svc-mooncake-demo-master`，而不是 Service 名。CacheRuntimeClass 中声明的是 headless Service，其本身不暴露端口，需要通过 Pod 的 DNS 记录访问。 +> 本示例使用 master Pod 的稳定 DNS 名 `mooncake-demo-master-0.svc-mooncake-demo-master` 作为连接地址，从而固定访问某一个副本。这里用 Service 名 `svc-mooncake-demo-master` 同样可行：Fluid 创建的组件 Service 是 headless 的（`clusterIP: None`）且未声明 `ports`，但这只影响 SRV 记录，A 记录仍会直接解析到后端 Pod IP，客户端随后直连容器的 `8080` 和 `50051` 端口。之所以推荐使用 Pod 级 DNS 名，是因为它在 master 扩容后仍然固定指向单个副本。 `setup()` 的输出（截取关键部分）： @@ -352,7 +365,7 @@ spec: nodeName: fluid-mooncake-worker containers: - name: client - image: btxu/mooncake:v3 + image: btxu/mooncake:v3@sha256:067614b70d25b496e3edc3480747d558ee8a364ef47a67f669f5d96ca5098552 imagePullPolicy: IfNotPresent command: ["sleep", "infinity"] env: @@ -369,7 +382,13 @@ mooncake-client-1 1/1 Running 0 3m44s 10.244.1.8 fluid-moo mooncake-client-2 1/1 Running 0 79s 10.244.2.15 fluid-mooncake-worker ``` -在第二个 Pod 中只做读取和校验（`setup` 参数与上文相同）： +第二个 Pod 中是一个全新的 Python 进程，需要先进入它的 Python 环境，并重复上文的导入语句和完整的 `MooncakeDistributedStore` 初始化（`setup` 参数与上文完全相同，`POD_IP` 同样取当前 Pod 自身的 IP）： + +```shell +$ kubectl exec -it mooncake-client-2 -- python3 +``` + +`store` 和 `hashlib` 初始化完成后，再做读取和校验： ```python got = store.get("demo_key") @@ -455,4 +474,4 @@ $ kubectl delete -f mooncake-client-2.yaml $ kubectl delete -f mooncake-client.yaml $ kubectl delete -f mooncake-dataset-runtime.yaml $ kubectl delete -f mooncake-cacheruntimeclass.yaml -``` \ No newline at end of file +``` diff --git a/samples/mooncake/docker/Dockerfile b/samples/mooncake/docker/Dockerfile new file mode 100644 index 00000000000..a3928e5ae53 --- /dev/null +++ b/samples/mooncake/docker/Dockerfile @@ -0,0 +1,27 @@ +# Build context for the image used by the Mooncake CacheRuntime sample. +# +# See docs/en/samples/cacheruntime/mooncake_cache_runtime.md +# docs/zh/samples/cacheruntime/mooncake_cache_runtime.md +# +# Build and push to a registry you control: +# docker build -t /mooncake:v3 samples/mooncake/docker +# docker push /mooncake:v3 + +FROM python:3.12-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2 curl jq \ + && rm -rf /var/lib/apt/lists/* + +RUN pip install --no-cache-dir \ + mooncake-transfer-engine-non-cuda==0.3.12.post1 \ + nvidia-cuda-runtime-cu12 + +ENV LD_LIBRARY_PATH=/usr/local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH + +# master rpc / worker data / metadata / master metrics / worker http +EXPOSE 50051 50052 8080 9003 9300 + +COPY custom-entrypoint.sh /custom-entrypoint.sh +COPY reportSummary.sh /reportSummary.sh +RUN chmod +x /custom-entrypoint.sh /reportSummary.sh diff --git a/samples/mooncake/docker/custom-entrypoint.sh b/samples/mooncake/docker/custom-entrypoint.sh new file mode 100755 index 00000000000..7402858a3fd --- /dev/null +++ b/samples/mooncake/docker/custom-entrypoint.sh @@ -0,0 +1,73 @@ +#!/bin/sh +# Component entrypoint invoked by Fluid's CacheRuntime. +# Usage: /custom-entrypoint.sh start + +set -e + +ROLE="$1" +ACTION="$2" + +if [ "$ACTION" != "start" ]; then + echo "Error: unsupported action '$ACTION'" + exit 1 +fi + +case "$ROLE" in + + master) + exec mooncake_master \ + -v=1 \ + --rpc_interface=eth0 \ + --enable_http_metadata_server=true \ + --http_metadata_server_host=0.0.0.0 \ + --http_metadata_server_port=8080 \ + --enable_metadata_cleanup_on_timeout=true \ + --client_ttl=10 + ;; + + worker) + # Read the runtime config JSON that Fluid mounts into the component pod. + if [ -z "$FLUID_RUNTIME_CONFIG_PATH" ] || [ ! -f "$FLUID_RUNTIME_CONFIG_PATH" ]; then + echo "Error: FLUID_RUNTIME_CONFIG_PATH not set or file not found" + exit 1 + fi + + CONFIG=$(cat "$FLUID_RUNTIME_CONFIG_PATH") + + MASTER_SVC=$(echo "$CONFIG" | jq -r '.master.service.name') + WORKER_SVC=$(echo "$CONFIG" | jq -r '.worker.service.name') + QUOTA=$(echo "$CONFIG" | jq -r '.worker.tieredStoreLevels[0].quotas[0] // "1GiB"') + + # Fluid reports the quota in Kubernetes units ("1Gi"); Mooncake expects "1GB". + SEGMENT_SIZE=$(echo "$QUOTA" | sed 's/Gi$/GB/; s/Mi$/MB/') + + NAMESPACE="${FLUID_DATASET_NAMESPACE:-default}" + MASTER_ADDR="${MASTER_SVC}.${NAMESPACE}.svc.cluster.local:50051" + METADATA_ADDR="http://${MASTER_SVC}.${NAMESPACE}.svc.cluster.local:8080/metadata" + WORKER_HOST="${POD_NAME}.${WORKER_SVC}.${NAMESPACE}.svc.cluster.local" + + echo "Starting worker: master=$MASTER_ADDR, segment_size=$SEGMENT_SIZE, host=$WORKER_HOST" + + exec mooncake_client \ + --host="$WORKER_HOST" \ + --port=50052 \ + --global_segment_size="$SEGMENT_SIZE" \ + --master_server_address="$MASTER_ADDR" \ + --metadata_server="$METADATA_ADDR" \ + --protocol=tcp \ + --enable_http_server=true \ + --http_port=9300 + ;; + + client) + # Mooncake is a client-less cache system in Fluid: applications link the + # Mooncake client library directly, so no client component is deployed. + echo "Error: client role is not applicable to Mooncake" + exit 1 + ;; + + *) + echo "Error: unknown role '$ROLE'" + exit 1 + ;; +esac diff --git a/samples/mooncake/docker/reportSummary.sh b/samples/mooncake/docker/reportSummary.sh new file mode 100755 index 00000000000..f360f7d47bf --- /dev/null +++ b/samples/mooncake/docker/reportSummary.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# ReportSummary entrypoint invoked by Fluid's CacheRuntime. It samples the +# Mooncake master's metrics endpoint and prints the JSON that Fluid writes into +# the Dataset's status.cacheStates field. +set -euo pipefail + +RAW=$(curl -s http://localhost:9003/metrics/summary) + +if [ -z "$RAW" ]; then + echo "Error: empty response from metrics endpoint" >&2 + exit 1 +fi + +# Extract the "Mem Storage: 0 B / 2.00 GB (0.0%)" segment. +MEM_LINE=$(echo "$RAW" | grep -oE 'Mem Storage: [^|]+' || true) + +CACHED_RAW=$(echo "$MEM_LINE" | sed -E 's/Mem Storage: ([^/]+) \/.*/\1/' | xargs) +CAPACITY_RAW=$(echo "$MEM_LINE" | sed -E 's/.*\/ ([^(]+) \(.*/\1/' | xargs) +PERCENT_RAW=$(echo "$MEM_LINE" | grep -oE '\([0-9.]+%\)' | tr -d '()%') + +# Normalize units ("2.00 GB" -> "2.00GiB"). +normalize_unit() { + echo "$1" | sed -E 's/ ?GB$/GiB/; s/ ?MB$/MiB/; s/ ?B$/B/' | tr -d ' ' +} +CACHED=$(normalize_unit "$CACHED_RAW") +CACHE_CAPACITY=$(normalize_unit "$CAPACITY_RAW") + +# Report the number of keys as fileNum. +FILE_NUM=$(echo "$RAW" | grep -oE 'Keys: [0-9]+' | grep -oE '[0-9]+' || echo "0") + +# Approximate the hit ratio with the success rate of Get requests. +GET_STATS=$(echo "$RAW" | grep -oE 'Get=[0-9.]+/[0-9.]+' || echo "Get=0.00/0.00") +GET_SUCCESS=$(echo "$GET_STATS" | cut -d= -f2 | cut -d/ -f1) +GET_TOTAL=$(echo "$GET_STATS" | cut -d/ -f2) +HIT_RATIO=$(awk -v s="$GET_SUCCESS" -v t="$GET_TOTAL" \ + 'BEGIN{ if (t>0) printf "%.0f", (s/t*100); else print "0" }') + +# Mooncake has no underlying storage, so report the cache capacity as ufsTotal. +UFS_TOTAL="$CACHE_CAPACITY" + +cat < Date: Fri, 28 Aug 2026 11:26:36 +0900 Subject: [PATCH 3/3] docs: address SonarCloud hotspots in the Mooncake sample image Install Python wheels only so no setup script runs at install time, pin nvidia-cuda-runtime-cu12, and run the components as an unprivileged user. Document why the worker's metadata and transfer connections stay clear-text. Signed-off-by: btxu-db --- samples/mooncake/docker/Dockerfile | 12 ++++++++++-- samples/mooncake/docker/custom-entrypoint.sh | 8 ++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/samples/mooncake/docker/Dockerfile b/samples/mooncake/docker/Dockerfile index a3928e5ae53..97ace579e19 100644 --- a/samples/mooncake/docker/Dockerfile +++ b/samples/mooncake/docker/Dockerfile @@ -13,9 +13,11 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2 curl jq \ && rm -rf /var/lib/apt/lists/* -RUN pip install --no-cache-dir \ +# Install from wheels only (--only-binary) so no package build/setup script is +# executed at install time, and pin every resolved version explicitly. +RUN pip install --no-cache-dir --only-binary=:all: \ mooncake-transfer-engine-non-cuda==0.3.12.post1 \ - nvidia-cuda-runtime-cu12 + nvidia-cuda-runtime-cu12==12.8.90 ENV LD_LIBRARY_PATH=/usr/local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH @@ -25,3 +27,9 @@ EXPOSE 50051 50052 8080 9003 9300 COPY custom-entrypoint.sh /custom-entrypoint.sh COPY reportSummary.sh /reportSummary.sh RUN chmod +x /custom-entrypoint.sh /reportSummary.sh + +# Run the components as an unprivileged user. All listening ports are above +# 1024 and the sample writes nothing to the container filesystem, so no root +# capability is required at runtime. +RUN useradd --create-home --uid 10001 --user-group mooncake +USER 10001 diff --git a/samples/mooncake/docker/custom-entrypoint.sh b/samples/mooncake/docker/custom-entrypoint.sh index 7402858a3fd..ae157867750 100755 --- a/samples/mooncake/docker/custom-entrypoint.sh +++ b/samples/mooncake/docker/custom-entrypoint.sh @@ -48,6 +48,14 @@ case "$ROLE" in echo "Starting worker: master=$MASTER_ADDR, segment_size=$SEGMENT_SIZE, host=$WORKER_HOST" + # The metadata endpoint is plain HTTP and the transfer protocol is TCP + # because Mooncake exposes no TLS variant for either: the master's built-in + # metadata server (--enable_http_metadata_server) only speaks HTTP, and the + # transfer engine only offers "tcp" and "rdma". Both connections stay inside + # the cluster, addressed by ClusterIP/headless service DNS, and carry cache + # blocks between components of this runtime only. Put the runtime in a + # dedicated namespace with a NetworkPolicy, or a service mesh with mTLS, if + # that traffic needs to be protected on the wire. exec mooncake_client \ --host="$WORKER_HOST" \ --port=50052 \