Increase gzip compression level to 9 (BestCompression)

[lyoung-confluent]

Currently when building an image apko uses pgzip.NewWriter which will use the DefaultCompression constant (-1) which is becomes a compression level of 5. We can decrease the final image/layer size at the cost of small amount of additional CPU during build time by instead opting into the BestCompression level (9) via pgzip.NewWriterLevel instead.

This won't necessarily result in a huge change in image size for most images, for example using examples/alpine-base.yaml it decreases from 27691520 bytes to 26865152 bytes (~3%).

[jonjohnsonjr]

I think this would be great to add as an option/flag, but I'm reluctant to make it the default. We want building (and rebuilding) images to be fast during development so we can get a fast feedback loop. Since melange uses apko to produce the build environment for our package builds, the vast majority of apko-produced images are effectively "single use" so it would be wasteful to burn the additional CPU to make them smaller, but it's definitely unfortunate that we're leaving some gains on the table for the final release images.

(Arguably, it might make more sense to have 9 be the default and have a flag to go faster during development...)

I think klauspost's choice for default strikes a good balance (some older discussion here), but I also spent a month last year chasing performance as much as possible, so I might have gone a little too far :)

Would you be okay with this as a flag?

[xnox]

To win on both speed, and image size.... We should consider Zstd compression. But I have no idea if the ecosystem of where images are executed can support that.

[jonjohnsonjr]

But I have no idea if the ecosystem of where images are executed can support that.

I feel like the major runtimes (containerd, mostly) probably already support zstd, but I'd be a little worried about things like Cloud Run. Would be worth experimenting with.

[xnox]

@jonjohnsonjr whilst the ephemeral containers are indeed single use; the ones that are published for production; pulled thousands of times and deployed thousands of times - are not ephemeral. And there we do win from best compression once; and smaller transfer and faster uncompress on the other end. Also production images are often compared on compressed size. Thus improving this might be a win.

[xnox]

Cloud Run accepts container images in the Docker Image Manifest V2, Schema 1, Schema 2, and OCI image formats. Cloud Run also accepts Zstd compressed container images.

https://docs.cloud.google.com/run/docs/container-contract

[xnox]

possibly we would need to do dual publication of gzip + zstd like fedora is pushing https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712/44

[coreydaley-cg]

Closing per the 2024 discussion: the default stays as it is for build-loop speed, and the thread's direction moved to zstd. #2376 adds a --compression flag, which is where a gzip level option belongs if someone wants one. Thanks for the measurement.