Design: ModelCache sources and the mount contract

[dennis-upbound]

Description of your changes

A ModelCache reads from HuggingFace and nowhere else, stages what it reads onto a ReadWriteMany volume on every matched cluster, and publishes a phase. A consumer works out how to read what was staged by deriving the volume's name — a derivation compose-model-replica and compose-model-cache each hold a copy of, with a comment telling both to change together, and which is true for one source.

This proposes two more sources and one status field.

source: OCI reads an artifact where it already is. The kubelet pulls the reference at pod start, mounts it read-only, and serves the next pod on that node from what the first pulled. Modelplane composes nothing per cluster for it. source: Existing mounts a claim the caller populated, which the same machinery makes nearly free.

status.clusters[].mount says how to read the artifact on that cluster — volumes, mounts and environment, in corev1's own types. A consumer copies the three lists without reading them. What does the mounting depends on what the artifact is: Kubernetes mounts a container image as an image volume, and a model artifact, which containerd will not mount, is read by a CSI driver. Modelplane resolves the reference once to tell them apart, so a reference it cannot serve becomes a Failed cluster rather than an empty directory at pod start.

What changed since the last look

The design now carries the evidence from Pablo's benchmark work, which arrived after it was drafted and sharpened the argument from consistency to correctness:

• Engine starts before its ModelCache has hydrated, so the first replica downloads the model itself #495 — an engine started while the cache was staging and downloaded the same 464 GB model in parallel, both finishing in the same second. Nothing gates on hydration: compose-model-deployment reads status.cache.storageClassName, which says a cluster has storage, never that the artifact is on it. §5.6 is what stops that.
• Nebius model cache never hydrates when its first node is replaced during cluster startup #494 — a replaced node stranded the staging volume, 40 minutes in Hydrating. GKE ModelCache PVC never binds: Filestore CSI driver addon not enabled on the cluster #221 is the same fault on GKE. §4.4 treats it as a class: each source that stages to a volume has it; a source that mounts from a registry does not.
• Model load from the EFS-backed ModelCache is slow #204, ModelCache sizeGiB has no effect below 1TiB on GKE: managed StorageClass is Filestore Enterprise #414, Design: shared model storage, define the bar before committing to an RWX backend #383 — the measured cost of the ReadWriteMany requirement, in §4.2.
• Deleting an InferenceCluster removes the GPU driver before its workloads, leaving stuck pods and reserved GPUs #499 — a new limitation. A CSI driver is needed to unmount, not only to mount, so this design puts a second component in the teardown path with the exposure Deleting an InferenceCluster removes the GPU driver before its workloads, leaving stuck pods and reserved GPUs #499 found on the DRA driver. Recorded rather than solved; the correction belongs to Deleting an InferenceCluster removes the GPU driver before its workloads, leaving stuck pods and reserved GPUs #499.

§6 is new, and is the part I'd most like a view on. This design argues OCI delivery wins on warm pulls — 11.7s against 40.7 minutes — and Modelplane publishes no series that measures either number. §6.1 proposes the two that make the claim falsifiable, with a cached label, because warm and cold differ by two orders of magnitude and one histogram holding both describes neither. §6.2 proposes four for cache state and marks them blocked by #476, where I've posted the research on how that group should be built.

On scope

This is the design alone. The implementation on dennis/oci-source-poc predates it and stages through modctl onto the cache volume — the approach this design replaces — so it would only confuse a review of the proposal. It needs rewriting against whatever lands here.

The doc supersedes #362 and #210, and revises design/modelcache.md, which will contradict it until that page is updated.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes. Design only; no function changes.
• Signed off every commit with git commit -s.

🤖 Generated with Claude Code

[dennis-upbound]

Closing until the design is reviewed in upbound/inference#7, which holds this document and the Upbound Inference one together. The split between them is the thing worth a reviewer's judgement, and it is only visible with both side by side.

This reopens, or comes back as a fresh PR, once that review settles. The branch stays at dennis/design-modelcache-sources.

Nothing here is blocked: #504 (the telemetry guide correction) and the research on #476 are independent.