oci-delta

oci-delta is a tool to take two oci archive files, called the "old" and "new" image below, containing bootc images and producing a resulting file, called a "delta" that can be used to update a bootc host with the old image installed to the new image, without having the new oci archive available. The advantage of using the delta is that it is significantly smaller, as it avoids shipping data that is already locally available from the installed old image.

Rust library

The Rust crate provides storage-independent delta parsing and layer reconstruction for consumers such as composefs-rs and bootc. It does not create deltas; use the Go CLI for that.

Mode of operation

An OCI image (and thus OCI archive) consists of some json metadata, and a list of compressed tar files, one for each image layer. Each layer is referenced from the metadata twice, once (the digest id) by sha256 digest of the compressed tar file, and once (the diff id) by the sha256 digest of the uncompressed file. The latter is important because various operations can cause layers to be recompressed, but using the diff_id we can ensure we're referencing the same data.

When bootc installs a new image (both when pulling from a registry or from an OCI archive), it will look at each layers digest id and diff_id comparing it to the set of already installed layers. If there is a match, the layer in the image isn't even looked at.

This allows the first level of deltas in oci-delta, we just generate a normal OCI image that leaves out the tar files for the layers that will already be installed. Such an oci archive can be installed with a command like bootc switch --transport=oci-archive $FILE.

The above is helped by the layer chunking that is typically done for bootc images. At the build time bootc tries to find related files (typically those installed from the same package), and puts them in separate layers. If two different images install the same package version they are likely to end up with an identical layer, even if the images are not directly related. This allows the first level of delta support to be more efficient than you would otherwise think.

The second level of delta is at the file level inside each layer. Even if a layer has changed, often many files are identical, and others are similar to the previous version of the same file. The assumption we make is that the target system has the complete old image locally available in some form, so we can use the entire old image as source material for the deltas. So, the idea is to create a delta format that allows reconstructing the "delta level 1" oci archive file given the information in the delta and the locally available old image. Then the reconstructed oci archive can be used with bootc switch to install the image.

To create these layer deltas, we use the tar-diff tool that creates binary deltas between tar files. Each changed layer of the new image is diffed against all the layers of the old image at once, with OCI whiteouts applied, so that the delta sources correspond to the merged filesystem of the old image rather than to individual layers.

Files in the source are addressed by their regular filesystem path, like usr/bin/bash. This matters because a typical bootc layer has content that looks something like this:

-rwxr-xr-x 0/0         1410656 1970-01-01 01:00 sysroot/ostree/repo/objects/8a/5d...d.file
hrwxr-xr-x 0/0               0 1970-01-01 01:00 usr/bin/bash link to sysroot/ostree/repo/objects/8a/5d...d.file

In other words, each file appears twice: once as an ostree object file, and once as a hardlink to it with the deployed path. We tell tar-diff to ignore the sysroot/ostree/ prefix when picking source names, so the deployed path is used instead. The ostree object paths are an artifact of how the image is packaged for installation; what is actually available on the target system afterwards is the merged image content addressed by deployed path.

At apply time, oci-delta needs to look up old file content by path. There are three ways to provide this:

• --ostree-repo (default, /ostree/repo): the ostree ref of the old image is located by matching the source image config digest recorded in the delta, and its commit is enumerated to map each deployed path to the corresponding repo object file.
• --container-storage: the old image is found in a podman container storage by config digest and mounted, and the mount is used as the source.
• --directory: a plain directory holding the old image content, i.e. an unpacked root filesystem, mostly useful for testing and for applying deltas off-system.

To completely support what is required, tar-diff supports:

• support the hardlinked structure of bootc images
• support multiple "old" layers (we use all old layers as delta source material)
• whiteout processing, so the sources reflect the merged old image
• ignoring delta source names by prefix

Of course, some layers are bound to be completely new, so we only store the tar-diff for layers where the diff is smaller than the original layer file.

There is one problem with this approach. The deltas (by their nature) work on the uncompressed layer tar files, so we have to recompress after reconstruction. This means the reconstructed image will have a manifest with different digest ids for the delta layers. The diff_id of the layers will be the same though, so bootc (at least a recent enough version) will be able to know that these layers are the same.

Example usage

Create a bootc delta from two oci archive.

$ oci-delta create old.oci-archive new.oci-archive update.oci-delta

Apply a delta on a bootc system.

$ oci-delta apply update.oci-delta reconstructed.oci-archive
$ bootc switch --transport=oci-archive reconstructed.oci-archive
$ rm update.oci-delta reconstructed.oci-archive

Delta sizes

Here are some example images and deltas between them. image is an automotive image, and image2 is a similar build that adds a file and an extra package. oldimage is similar to base, but its older so it will have older versions of some packages.

-rw-r--r--. 1 alex alex 318M 27 mar 11.30 oldimage.oci-archive
-rw-r--r--. 1 alex alex 306M 27 mar 14.25 image.oci-archive
-rw-r--r--. 1 alex alex 309M 27 mar 14.25 image2.oci-archive

Here are the delta sizes going between these.

-rw-r--r--. 1 alex alex  21M 27 mar 14.28 oldimage-to-image.delta
-rw-r--r--. 1 alex alex  16M 27 mar 14.30 image-to-image2.delta

I also computed a full delta between quay.io/fedora/fedora-bootc:41-x86_64 and quay.io/fedora/fedora-bootc:42-x86_64:

-rw-r--r--. 1 alex alex 972M 30 mar 11.37 fedora-bootc-41-x86_64.oci-archive
-rw-r--r--. 1 alex alex 999M 30 mar 11.37 fedora-bootc-42-x86_64.oci-archive
-rw-r--r--. 1 alex alex 555M 30 mar 11.56 fedora-41-to-42.delta

There is a script tools/analyze-delta.py in the repo that analyzes the produced deltas (with more detail if you pass --verbose). This allows you to see in more detail what is being reused.

E2E Testing

End-to-end tests validate the full delta create/apply/bootc-switch workflow using real Fedora bootc images.

make test-e2e

The test auto-detects the environment:

• VM mode (requires KVM + libvirt): builds a bootc VM, creates and applies a delta inside it, reboots, and verifies the new deployment with bootc status.
• Container mode (fallback): runs in a privileged container. Validates delta create/apply and image integrity, but cannot verify the reboot cycle.

Override the test images or mode with environment variables:

BASE_IMAGE=quay.io/fedora/fedora-bootc:43 TARGET_IMAGE=quay.io/fedora/fedora-bootc:44 TEST_MODE=vm make test-e2e

VM mode requires sudo access for bootc-image-builder and libvirt operations.

CI runs E2E tests on Testing Farm via Packit (.packit.yaml), which provisions KVM-capable infrastructure for full VM-mode testing.

Requirements

The target system has to run bootc 1.15.0 or later, which matches reconstructed layers by diff_id.

Delta file format

A delta file is an uncompressed tar archive containing an OCI image layout, where the single manifest describes a delta artifact rather than a runnable image. This is an oci-archive form that matches what would normally be used for OCI artifacts, and the delta could indeed be stored in a registry as an oci artifact.

This format is very similar to the one described in https://github.com/flatpak/flatpak-oci-specs, but with these changes:

• The delta manifest uses artifactType instead of a custom config mimetype.
• We store the original manifests as layers (to make the delta applicable stand-alone).
• Delta layer order doesn't exactly match the original "new" image, and all layers need not be available.
• Layers that are used as-is from the original "old" image are recorded in a new delta.reused annotation.
• We allow layers to be stored as the original tar.gz (if delta was not helpful for that layer).
• The delta.from layer annotation is no longer used, as we diff against all layers in source image.
• We support embedding cosign signatures

Outer structure

oci-layout
index.json                          - points to the delta manifest
blobs/sha256/<delta-manifest-hash>  - the delta manifest
blobs/sha256/<delta-config-hash>    - empty config object "{}"
blobs/sha256/<image-manifest-hash>  - the target image manifest (embedded)
blobs/sha256/<image-config-hash>    - the target image config (embedded)
blobs/sha256/<layer-data-hash>      - one blob per changed layer (tar-diff or original gzip)
blobs/sha256/<sig-manifest-hash>    - (optional) cosign signature manifest
blobs/sha256/<sig-config-hash>      - (optional) cosign signature config
blobs/sha256/<sig-payload-hash>     - (optional) cosign signature payload layer(s)

Delta manifest

The delta manifest is a standard OCI image manifest with artifactType set to application/vnd.io.github.containers.oci-delta.v1 and an empty config (application/vnd.oci.empty.v1+json). The subject field references the target image manifest, enabling referrer discovery when the delta is stored in a registry.

It has the following top-level annotations:

Annotation	Description
io.github.containers.delta.target	Digest of the target image manifest
io.github.containers.delta.source	Digest of the old image manifest
io.github.containers.delta.source-config	Digest of the old image config
io.github.containers.delta.reused	JSON array of layer digests that are expected to already be present on the target system
io.github.containers.delta.reused-diff-id	JSON array of diff_ids (uncompressed sha256) for the reused layers, parallel to delta.reused

Example delta manifest:

{
  "schemaVersion": 2,
  "artifactType": "application/vnd.io.github.containers.oci-delta.v1",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b367...",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.io.github.containers.oci-delta.image-manifest.v1",
      "digest": "sha256:867b4193920b8192...",
      "size": 715,
      "annotations": {
        "io.github.containers.delta.content": "image-manifest"
      }
    },
    {
      "mediaType": "application/vnd.io.github.containers.oci-delta.image-config.v1",
      "digest": "sha256:49dc3229d8ed7b49...",
      "size": 299,
      "annotations": {
        "io.github.containers.delta.content": "image-config"
      }
    },
    {
      "mediaType": "application/vnd.tar-diff",
      "digest": "sha256:453b6a6f17f0ab18...",
      "size": 674,
      "annotations": {
        "io.github.containers.delta.content": "image-layer",
        "io.github.containers.delta.to": "sha256:dc2c7d87dce684ab..."
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:48b5bc4b742a2...",
      "size": 9822,
      "annotations": {
        "io.github.containers.delta.content": "cosign-signature"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:38c384ddf6673...",
      "size": 233,
      "annotations": {
        "io.github.containers.delta.content": "cosign-signature-content"
      }
    },
    {
      "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
      "digest": "sha256:41e82de5226ac...",
      "size": 241,
      "annotations": {
        "io.github.containers.delta.content": "cosign-signature-content"
      }
    }
  ],
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:867b4193920b8192...",
    "size": 715
  },
  "annotations": {
    "io.github.containers.delta.reused": "[\"sha256:ba4dfb86...\",\"sha256:cfa55bab...\"]",
    "io.github.containers.delta.reused-diff-id": "[\"sha256:a8321da4...\",\"sha256:4f6932a5...\"]",
    "io.github.containers.delta.source": "sha256:cf1de3f5d45cff6b...",
    "io.github.containers.delta.source-config": "sha256:88e577dc6d4ab25c...",
    "io.github.containers.delta.target": "sha256:867b4193920b8192..."
  }
}

Delta manifest layers

Every layer in the delta manifest carries an io.github.containers.delta.content annotation that identifies its role. Parsers should dispatch on this annotation and ignore layers with unknown content values for forward compatibility.

delta.content value	Description
image-manifest	The complete manifest of the target image
image-config	The complete config of the target image (contains diff_ids)
image-layer	A changed layer — either a tar-diff delta or the original gzip layer
cosign-signature	A cosign/sigstore signature manifest (optional)
cosign-signature-content	A blob referenced by a cosign signature manifest (optional)

Image manifest (image-manifest): needed during apply to know the full layer list and reconstruct the output OCI archive.

Image config (image-config): needed to obtain the diff_ids for validating reconstructed layers.

Image layers (image-layer): one per changed layer, with an additional io.github.containers.delta.to annotation containing the digest of the target layer this blob reconstructs. Has one of two media types:

• application/vnd.tar-diff: a tar-diff binary delta. Applying it against the locally available content of the old image produces the uncompressed target layer tar. The result must be gzip-compressed and its diff_id validated before use.
• application/vnd.oci.image.layer.v1.tar+gzip: the original layer, stored verbatim when the tar-diff would have been larger.

Signature layers (cosign-signature, cosign-signature-content): optional. These allow signature verification at import time without registry access. Each signature artifact adds three entries: the signature manifest (cosign-signature), plus its config and payload blobs (cosign-signature-content). The signature manifest blob contains the full cosign layer descriptors with their original annotations. The config and payload blobs are listed in the delta manifest to ensure they are preserved when the delta is copied between storage backends.

Layers whose diff_id matches one already installed on the system (listed in delta.reused-diff-id) are absent from the delta entirely. The apply tool omits them from the reconstructed archive and relies on bootc's diff_id-based deduplication to find them locally.

Applying a delta

To reconstruct a usable OCI archive from a delta:

1. Parse the delta manifest and locate the embedded image manifest and config by io.github.containers.delta.content annotations.
2. For each layer in the image manifest, find the matching delta layer by delta.to annotation.
   • If found as a tar-diff: apply it against the local copy of the old image (looked up by the delta.source-config digest in the ostree repo or container storage, or given explicitly as a directory), resolving source files by their deployed path, then gzip-compress the result and verify the diff_id matches the config.
   • If found as original gzip: copy the blob directly.
   • If not found: the layer is reused - omit the blob from the output (bootc will locate it by diff_id).
3. Write a new OCI archive with the image config, the reconstructed layer blobs, and a rewritten image manifest reflecting the new digests of any recompressed layers.

Embedded signatures

Deltas can optionally carry cosign/sigstore signatures. This enables signature verification during import without requiring registry access (important for offline update scenarios).

The signature artifact is stored as-is in the delta's blob directory, with its manifest, config, and payload layers all listed in the delta manifest. The signature manifest blob contains the full cosign layer descriptors with their original annotations (the actual cryptographic signature, Fulcio certificate, certificate chain, and Rekor transparency log bundle).

At import time, the consumer can extract the signature manifest from layers annotated with io.github.containers.delta.content of cosign-signature and cosign-signature-content, then verify the signature against the embedded image manifest digest using the appropriate policy and keys.