In Kubernetes driver, how can I reduce the memory (docker layer cache) within a Pod?

[githubeto]

I am using ECR as the storage and usage destination for caches with the Kubernetes driver.
However, after a build without cache is executed, I have confirmed that some memory within the Pod remains occupied and does not get freed up.
Since the memory within the node continues to be used for an extended period, I am looking to reduce memory usage outside of build execution.

example.

docker buildx build . \
  -t xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest  \
  --cache-to compression=zstd,mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest \
  --cache-from type=registry,ref=xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest \
  --push

This issue seems to occur not only because the cache is pushed to ECR but also because the layer cache remains locally, and I have confirmed that it can be released by executing docker buildx prune --builder <builder-name>.

I looked into the GC documentation, and while it seems possible to adjust the timing, I couldn't find a setting to not leave the cache locally.

https://docs.docker.com/build/cache/garbage-collection/

When using ECR's Cache with the Kubernetes driver, what should I do about the local cache? Is setting the GC timing to a very short interval the only option, or is there another approach?

[wellkilo]

Registry cache is an additional export; it does not make the builder stateless. BuildKit still retains local records/blobs for the active build and reuse. --no-cache disables cache lookup for that build, but it does not mean “retain nothing afterward.”

For a long-lived Kubernetes builder, configure BuildKit garbage collection with a small disk budget/short keep duration, and inspect/prune explicitly:

docker buildx du --builder <builder-name>
docker buildx prune --builder <builder-name> --filter 'until=1h' --force

Mount a buildkitd.toml with your GC policies into the Kubernetes driver pods. There is no registry-cache option that disables all local caching while keeping a long-lived builder.

If you need near-zero idle footprint, use ephemeral builder pods: create/bootstrap the builder for the build, push the image and registry cache, then remove the builder. That trades startup time and local reuse for cleanup certainty.

Also distinguish memory from disk/page cache. buildx du reports BuildKit cache records on disk. Linux may retain freed file pages as reclaimable cache, so pod/node “used memory” need not fall immediately after buildx prune. Compare working set/RSS, filesystem usage, and buildx du rather than treating all cached memory as a leak.

Thus the operational choices are aggressive GC or ephemeral builders; exporting cache to ECR alone does not turn off the local cache.