Atomic pull tries to get signature files that don't exist off http sigstore

[ServerNinja]

My problem:

The issue is that when I do an sudo atomic pull, it tries to download signature-1 and then signature-2 but signature-2 was never generated when I previously did sudo atomic push.

I'm not sure what I'm doing wrong or if this is an actual bug. The sudo atomic pull command just tries to download signature-1, signature-2, signature-3, etc... even though sudo atomic push only generates a single signature-1 file.

Steps I took:

1. On image creation workstation: downloaded alpine image (as a test), re-tag and use sudo atomic push to push to atomic registry (running on a "stand-alone" openshift 3.9 docker registry configuration)

sudo docker pull alpine:latest
sudo docker tag alpine:latest docker-registry-default.common.xxx.com/test/alpine:latest
sudo atomic push docker-registry-default.common.xxx.com/test/alpine:latest --sign-by ops@xxx.com --gnupghome /home/vagrant/.gnupg

2. Copy signatures created to s3 bucket
3. On "destination" workstation, I log into docker and sudo atomic pull:

oc login https://atomic-registry01.common.xxx.com:8443 -u registryuser -p xxxxxxxxx
sudo docker login -u registryuser -p $(oc whoami -t) docker-registry-default.common.xxx.com

sudo atomic pull docker-registry-default.common.xxx.com/test/alpine
FATA[0004] Source image rejected: Error reading signature from https://s3.amazonaws.com/dis.xxx.com/test/alpine@sha256=cf2d5c15199a1f6161650f2cfe35fd502d0b661823030c7df599050ad895580e/signature-2: status 403

On Image Creation workstation:

/etc/containers/registries.d/*

cat /etc/containers/registries.d/docker-registry-default.common.xxx.com-test.yaml
docker:
  docker-registry-default.common.xxx.com/test:
    sigstore: https://s3.amazonaws.com/dis.xxx.com
    sigstore-staging: file:///home/vagrant/atomic-sigstore

cat /etc/containers/registries.d/default.yaml
# This is a default registries.d configuration file.  You may
# add to this file or create additional files in registries.d/.
#
# sigstore: indicates a location that is read and write
# sigstore-staging: indicates a location that is only for write
#
# sigstore and sigstore-staging take a value of the following:
#   sigstore:  {schema}://location
#
# For reading signatures, schema may be http, https, or file.
# For writing signatures, schema may only be file.

# This is the default signature write location for docker registries.
default-docker:
#  sigstore: file:///var/lib/atomic/sigstore
  sigstore-staging: file:///var/lib/atomic/sigstore

# The 'docker' indicator here is the start of the configuration
# for docker registries.
#
# docker:
#
#   privateregistry.com:
#    sigstore: http://privateregistry.com/sigstore/
#    sigstore-staging: /mnt/nfs/privateregistry/sigstore

policy.json

cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
            "docker-registry-default.common.xxx.com/test": [
                {
                    "keyType": "GPGKeys",
                    "type": "signedBy",
                    "keyData": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
                }
            ]
        },
        "docker-daemon": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }
    }
}

atomic.conf

cat /etc/atomic.conf
# Atomic CLI configuration file

default_scanner:
default_docker: docker
registry_confdir: /etc/containers/registries.d/
discover_sigstores: true
sigstore_metadata_image: sigstore


# Default storage backend [ostree, docker]
# default_storage: docker
# ostree_repository: /ostree/repo
# checkout_path: /var/lib/containers/atomic
#

# Default identity for signing images
# default_signer:
# Absolute path to GPG keyring. Value set as environment variable GNUPGHOME
#gnupg_homedir: /home/USER/.gnupg
#
# To always use a proxy with atomic, you can uncomment and fill out
# below.
#
#http_proxy:
#https_proxy:
#no_proxy:

On Destination workstation:

/etc/containers/policy.json

$ cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "docker-registry-default.common.xxx.com/test": [
                {
                    "keyType": "GPGKeys",
                    "type": "signedBy",
                    "keyPath": "/home/vagrant/docker_image_key.gpg"
                }
            ]
        },
        "docker-daemon": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }
    }
}

/etc/containers/registries.d/*

$ cat /etc/containers/registries.d/docker-registry-default.common.xxx.com-test.yaml
docker:
  docker-registry-default.common.xxx.com/test:
    sigstore: https://s3.amazonaws.com/dis.xxx.com

$ cat /etc/containers/registries.d/default.yaml
# This is a default registries.d configuration file.  You may
# add to this file or create additional files in registries.d/.
#
# sigstore: indicates a location that is read and write
# sigstore-staging: indicates a location that is only for write
#
# sigstore and sigstore-staging take a value of the following:
#   sigstore:  {schema}://location
#
# For reading signatures, schema may be http, https, or file.
# For writing signatures, schema may only be file.

# This is the default signature write location for docker registries.
default-docker:
#  sigstore: file:///var/lib/atomic/sigstore
  sigstore-staging: file:///var/lib/atomic/sigstore

# The 'docker' indicator here is the start of the configuration
# for docker registries.
#
# docker:
#
#   privateregistry.com:
#    sigstore: http://privateregistry.com/sigstore/
#    sigstore-staging: /mnt/nfs/privateregistry/sigstore

atomic.conf:

$ cat /etc/atomic.conf
# Atomic CLI configuration file

default_scanner:
default_docker: docker
registry_confdir: /etc/containers/registries.d/
discover_sigstores: false
sigstore_metadata_image: sigstore


# Default storage backend [ostree, docker]
# default_storage: docker
# ostree_repository: /ostree/repo
# checkout_path: /var/lib/containers/atomic
#

# Default identity for signing images
# default_signer:
# Absolute path to GPG keyring. Value set as environment variable GNUPGHOME
#gnupg_homedir: /home/USER/.gnupg
#
# To always use a proxy with atomic, you can uncomment and fill out
# below.
#
#http_proxy:
#https_proxy:
#no_proxy:

[ServerNinja]

I should also note the sudo atomic trust show on the destination workstation:

* (default)                         reject
docker-registry-default.common.xxx.com/test signed ops@xxx.com```

[ServerNinja]

$ sudo atomic --version
1.22.1

[ServerNinja]

Skopeo Version:

$ skopeo --version
skopeo version 0.1.31

[ServerNinja]

I wonder if the issue is that the skopeo command is a newer build than what the atomic command is expecting. In the debug statements, atomic is using out-dated syntax for skopeo...

FATA[0004] Source image rejected: Error reading signature from https://s3.amazonaws.com/dis.xxx.com/test/alpine@sha256=cf2d5c15199a1f6161650f2cfe35fd502d0b661823030c7df599050ad895580e/signature-2: status 403

Traceback (most recent call last):
  File "/bin/atomic", line 185, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 62, in pull_image
    be.pull_image(self.args.image, remote_image_obj, debug=self.args.debug, assumeyes=self.args.assumeyes, src_creds=src_creds)
  File "/usr/lib/python2.7/site-packages/Atomic/backends/_docker.py", line 348, in pull_image
    policy_filename=trust.policy_filename, src_creds=src_creds)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 454, in skopeo_copy
    return check_call(cmd, env=os.environ)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 214, in check_call
    return subprocess.check_call(cmd, env=env, stdin=stdin, stderr=stderr, stdout=stdout, close_fds=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/skopeo', '--policy=/etc/containers/policy.json', '--debug', 'copy', '--src-tls-verify=false', '--dest-tls-verify=false', '--remove-signatures', 'docker://docker-registry-default.common.xxx.com/test/alpine:latest', 'docker-daemon:docker-registry-default.common.xxx.com/test/alpine:latest']' returned non-zero exit status 1

I was able to manually run the following skopio command using the correct syntax and it works... so atomic is wrapping the skopio command wrong.

[vagrant@centos ~]$ skopeo --policy=/etc/containers/policy.json copy --src-tls-verify=false --dest-tls-verify=false --remove-signatures atomic:docker-registry-default.common.xxx.com/test/alpine:latest docker-daemon:docker-registry-default.common.xxx.com/test/alpine:latest
Copying blob sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde
 2.10 MB / 2.10 MB [========================================================] 3s
Copying config sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321
 1.48 KB / 1.48 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures

Also when I use the proper skopio command (that atomic fails to use), it actually pushes the image properly and stores the signatures in the atomic registry correctly...

The issue is that the atomic command refers to the atomic registry by putting atomic://docker-registry/image:latest in the command. The format skopio expects is: atomic:docker-registry/image:latest (instead of ://, just use :).

This need to be fixed.

Also it should be noted that the skopeo version I'm using as well as the atomic command I'm using was installed via YUM on a CentOS 7.5 box.