build: add support for zstd image compression

[milas]

Changes

A pretty mechanical plumbing of a new --compression flag through the various machinery.

The default is still gzip for backwards compatibility, but --compression=zstd can be specified as an alternative.

Container runtimes have good zstd support these days and it compresses better/with less CPU than gzip.

Context

Starting with google/go-containerregistry v0.21.8, it's possible to make images with correctly named ZSTD-compressed layers.

(Previously, it was possible to write zstd-compressed layer data, but it'd still be .tar.gz, which was confusing at best but more likely to break downstream tooling.)

See google/go-containerregistry#2382

The go.mod here already picked that up, so no dependency update is necessary.

[milas]

FYI I did a quick scan of other open issues/PRs -- I do see there's WIP for adding EROFS (very cool!) @ #2249, which I don't think really conflicts. Arguably, this can work together with that since EROFS layers can similarly be gzip/zstd/etc compressed.

[milas]

I've accumulated merge conflicts over the past month (e.g. I see #2479 recently changed compression stuff a fair amount).

Happy to rebase if there's any interest from maintainers here

[coreydaley-cg]

Coming back to this once #2479 lands. That PR folds the layer writers into one constructor where the compressor is a single dst choice, so this rebases onto a smaller and cleaner seam than the current compress() and newLayerWriter branches. Rebasing before it merges would just mean doing it twice.

Two asks for the rebase. First, the flag help should say that zstd layers need Docker 23.0 or newer to pull; registries don't care, but older clients can't decompress them, and that's the reason gzip stays the default. Second, some numbers to go with the change: for a couple of representative images, compression wall time and CPU, encoder memory with several layer writers open at once (layered builds hold every writer open through the filesystem walk, and zstd's window is heavier than pgzip's blocks), resulting layer sizes, and decompression time on pull. Decompression is where zstd should win clearly; the rest is what would justify the memory it costs at build time.