Sync up unique cache paths for Singularity images with cwl-utils

[adamnovak]

This and the corresponding change to cwl-utils cwl-docker-extract should solve the problem of ambiguous or different SIF file names between the two tools.

There's still the problem that cwltool supports more ways of specifying an image than cwl-utils does.

[codecov]

Codecov Report

❌ Patch coverage is 88.09524% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.30%. Comparing base (b4c828c) to head (34a0115).

Files with missing lines	Patch %	Lines
cwltool/singularity.py	88.09%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2284      +/-   ##
==========================================
+ Coverage   85.01%   85.30%   +0.28%     
==========================================
  Files          46       46              
  Lines        8621     8640      +19     
  Branches     2020     2023       +3     
==========================================
+ Hits         7329     7370      +41     
+ Misses        817      804      -13     
+ Partials      475      466       -9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[adamnovak]

The linter that complains about long lines has a different line length limit than, and runs before, the linter that tells you your long lines are not wrapped exactly correctly.

[adamnovak]

I'm having trouble running the offending test locally because I can't figure out how to limit the test system to a reasonable number of worker threads.

When I run tox -e py310-unit -- tests/test_singularity.py::test_singularity3_docker_image_id_in_tool on my Linux host it gets stuck after having started 130 of 160 worker processes, and I think it is probably trying to start too many.

[adamnovak]

Turns out I need:

tox -e py310-unit -- tests/test_singularity.py::test_singularity3_docker_image_id_in_tool -n3

[adamnovak]

OK, test_singularity3_docker_image_id_in_tool seems to be testing that you can run a workflow that asks for docker.io/debian:stable-slim, and then one in the same directory that asks for docker.io_debian:stable-slim.sif, assuming that we always replace / with exactly one _ in the image names and we can pick up the image by its relative SIF path.

Do we really need to promise that constraint?

We also have a similar test in test_singularity2_docker_image_id_in_tool that wants us to put docker.io/debian:stable-slim at debian:stable-slim.img, which I don't think we were doing before; I'm not sure the CI still attempts Singularity 2 testing.

[adamnovak]

@mr-c I'm not sure what to do about the coverage shortfall here. It looks like CI needs tests with (very old by now) Singularity 2, or it needs a test case where dockerImageId is used to point to a file and not a sandbox, or to the name of an image where the cached file exists already.

[adamnovak]

But we already run debian_image_id2.cwl where dockerImageId points to a .sif file. So maybe the coverage tool isn't picking it up?

[adamnovak]

The mypy tests are failing because they're somehow also actually running the unit tests, and the unit tests nondeterministically fail with things like:

  E   RuntimeError: jshint failed to run successfully
  E   returncode: 0
  E   stdout: ""
  E   stderr: "INFO:    Using cached SIF image
  E   INFO:    Mounting image with FUSE.
  E   /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-479582912/root: Operation not permitted
  E   FATAL:   While running host post start tasks: while unmounting fuse directory: /tmp/rootfs-479582912/root: exit status 1
  E   /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-479582912/root: Operation not permitted
  E   ERROR:   While running host cleanup tasks: while unmounting fuse directory: /tmp/rootfs-479582912/root: exit status 1
  E   ERROR:   Unprivileged host cleanup failed: host cleanup failed
  E   FATAL:   host post start process failed: host po"

[adamnovak]

This looks a lot like sylabs/singularity#4044 which was fixed in Sylabs Singularity 4.5.0. CI is running with 4.3.5.

[adamnovak]

Now a bunch of CWL 1.1 (only) Podman (only) conformance tests are failing, apparently because of:

Error: OCI runtime error: crun: unknown version specified

[adamnovak]

Now the CWL conformance (v1.3.0-dev1, --relax-path-checks) test failed because when it went to access the network the connection was reset.

[adamnovak]

The Python 3.10 mypy tests also failed because, when they went to actually run the tests (instead of just type checking), they failed to talk to the network for some of the tests where we're testing if we can use network URLs to CWL documents.

[adamnovak]

The Python 3.12 mypy tests failed with another instance of the unmount/cleanup failure:

  INFO:    Converting OCI blobs to SIF format
  INFO:    Starting build...
  INFO:    Fetching OCI image...
  INFO:    Extracting OCI image...
  INFO:    Inserting Singularity configuration...
  INFO:    Creating SIF file...
  WARNING: path /overlay-lowerdir/tmp is already overridden by /tmp/vv2ld6uu, replacing with /var/lib/singularity/mnt/session/tmp
  /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-2672171906/root: Operation not permitted
  FATAL:   While running host post start tasks: while unmounting fuse directory: /tmp/rootfs-2672171906/root: exit status 1
  /usr/local/bin/fusermount3: failed to unmount /tmp/rootfs-2672171906/root: Operation not permitted
  ERROR:   Failed to clean up FUSE mount: while unmounting fuse directory: /tmp/rootfs-2672171906/root: exit status 1
  ERROR:   Unprivileged host cleanup failed: host cleanup failed
  FATAL:   host post start process failed: host post start tasks failed
  ERROR:   While running host cleanup tasks: encountered errors during CleanupHost: [while unmounting fuse directory: /tmp/rootfs-2672171906/root: exit status 1]

So something like sylabs/singularity#4044 still exists.

I have singularity-ce version 3.10.3 and that seems to work fine, so we might want to downgrade?

[adamnovak]

I'm going to de-upgrade Singularity so I can't be accused of causing all these unmount failures.

[adamnovak]

I'm now getting Error: OCI runtime error: crun: unknown version specified errors for the CWL 1.2 Podman tests but not the CWL 1.1 Podman tests.

I'm going to rerun them.

[adamnovak]

@mr-c This and common-workflow-language/cwl-utils#433 are now passing tests, although I think that there are substantial preexisting problems with flaky tests that are not being solved here.

[adamnovak]

Per discussion at the CWL4HPC meeting, I'm going to adjust this and the cwl-utils PR so that, when an image is found at an old-style path and not at the new-style one, it gets hardlinked-or-copied to the new path.

Both cwl-utils and cwltool will do this.

The hardlink-or-copy widget (and as much other code as possible) should be in cwl-utils and imported in cwl-tool. I will set up a Git dependency link and @mr-c will handle turning that into a real set of interlinked releases.

[adamnovak]

@mr-c I can run through a few tests manually with this, but the only environment I have that could e.g. run singularity build is the CI environment, and it is not happy with my Git link in requirements.txt and is just not using it and instead using a different cwl-utils version.

Can you sort this out somehow?