tests/simple-validation-image-test.py: adjust to support base image tarballs

[kanavin]

With these commits, base image tarballs (generated from tests/base-extended/simple-validation/image-base.xml) can be validated (with 100% pass) using the validation script directly:

$ PYTHONPATH=. tests/simple-validation-image-test.py elbe-build-20260806-142454/

or with a higher level elbe command:

$ ./elbe check-build img elbe-build-20260806-142454/

[t-8ch]

f-strings would be fine.

[t-8ch]

We should also test that finetuning works in both the base and the extended image.
Maybe we can use the full simple-validation image to also emit a base image tarball and use that.
(And then also reuse the existing test fully for that and both the base and the extended variants)

We could skip the pbuilder bits with a variant to keep the speed up, though.

[kanavin]

We should also test that finetuning works in both the base and the extended image. Maybe we can use the full simple-validation image to also emit a base image tarball and use that. (And then also reuse the existing test fully for that and both the base and the extended variants)

That won't work because the full simple-validation image does things in finetuning (e.g. delete packaging data) that make the output unsuitable for making extended images.

[t-8ch]

The deletion of the packaging data is probably only cargo-culted from an example.
For this test we can just drop it. We have enough other tests for <command>.

[t-8ch]

And the <rm>var/cache/apt/archives/*.deb</rm> bit should be fine to keep.

[kanavin]

I think we need to step back and decide what should the top level test contain (e.g. elbepack/tests/test_xml.py). I have patches upcoming that modify the base-extended test to do this:

def test_base_extended_build(initvm, tmp_path):
...
     initvm('submit', '--output', base_build, '--skip-build-bin', '--skip-build-sources',
            base_xml_path)
    run_elbe_subcommand(['check-build', 'img', base_build])
     initvm('submit', '--output', extended_build, '--skip-build-bin', '--skip-build-sources',
            '--base-image', base_build_image, extended_xml_path)
    run_elbe_subcommand(['check-build', 'img', extended_build])

E.g. run the sequence:

• build base image from tests/base-extended/simple-validation/
• run validation script on it
• build extended image from tests/base-extended/simple-validation/
• run validation script and qemu bootup/login test on it.

The test for the existing classic validation image and the image itself aren't changed.

Should we complicate matters with also adding finetuning to those images and/or doing something with the original validation image? I'd rather improve tests incrementally. The goal for me, for now, is to make it possible to validate existing base-extended images and add that validation to existing tests for them.

[kanavin]

So how do we go from the above to actual implementation?

Here's the proposal:

• do not validate images in tests/base-extended/simple-validation/ with a validation script. Only do the qemu boot/login test on the base/extended image. That way the script doesn't have to be modified to fit them with all the complication and parameters that brings.
• add a tarball output to tests/simple-validation-image.xml
• add test_extended(initvm, simple_build) which will take the tarball output from simple_build (if source.xml specifies that there's a tarball, otherwise skip the test), build an extended image, and validate it with 'elbe check-build'
• the extended image definition is separate from one in tests/base-extended/simple-validation/ as it does need to include the validation script

Any tweaks to the above?

[t-8ch]

• do not validate images in tests/base-extended/simple-validation/ with a validation script. Only do the qemu boot/login test on the base/extended image. That way the script doesn't have to be modified to fit them with all the complication and parameters that brings.

I would have dropped the current image-base.xml completely. The boot test also doesn't make sense on the boot test.

• add test_extended(initvm, simple_build) which will take the tarball output from simple_build (if source.xml specifies that there's a tarball, otherwise skip the test), build an extended image, and validate it with 'elbe check-build'

When would the tarball not exist?

• the extended image definition is separate from one in tests/base-extended/simple-validation/ as it does need to include the validation script

Do we need two different extended images in the tests?

My idea was:

• Add a tarball output to simple-validation-image.xml,
  Test both sda.img and the base tarball with the test script.
  Only do the boot tests for sda.img.
• Add an extended image on top of simple-validation-image.xml.
  This extended image installs a package, does some light finetuning and an archive.
  The extended image is boot tests, and validated against the full simple-validation-image-test.py to make sure that everything is passed through and also test extended image specific bits.

[kanavin]

The issue is that build of simple-validation-image.xml isn't tested standalone. It's part of the 'test matrix' which takes available tests, images (obtained by making a list of tests/simple*.xml), and 'elbe check-build' options, and combines them in every possible way:

$ pytest --co
...
        <Function test_simple_build[simple-arm64.xml-schema]>
        <Function test_simple_build[simple-arm64.xml-cdrom]>
        <Function test_simple_build[simple-arm64.xml-img]>
        <Function test_simple_build[simple-arm64.xml-sdk]>
        <Function test_rebuild[simple-arm64.xml]>
        <Function test_check_updates[simple-arm64.xml]>
        <Function test_simple_build[simple-validation-image.xml-schema]>
        <Function test_simple_build[simple-validation-image.xml-cdrom]>
        <Function test_simple_build[simple-validation-image.xml-img]>
        <Function test_simple_build[simple-validation-image.xml-sdk]>
        <Function test_rebuild[simple-validation-image.xml]>
        <Function test_check_updates[simple-validation-image.xml]>
        <Function test_simple_build[simple-rescue-busybox-dyn-cpio.xml-schema]>
        <Function test_simple_build[simple-rescue-busybox-dyn-cpio.xml-cdrom]>
        <Function test_simple_build[simple-rescue-busybox-dyn-cpio.xml-img]>
        <Function test_simple_build[simple-rescue-busybox-dyn-cpio.xml-sdk]>
        <Function test_rebuild[simple-rescue-busybox-dyn-cpio.xml]>
        <Function test_check_updates[simple-rescue-busybox-dyn-cpio.xml]>
        <Function test_simple_build[simple-amd64-with-grub-uefi.xml-schema]>
        <Function test_simple_build[simple-amd64-with-grub-uefi.xml-cdrom]>
        <Function test_simple_build[simple-amd64-with-grub-uefi.xml-img]>
        <Function test_simple_build[simple-amd64-with-grub-uefi.xml-sdk]>
        <Function test_rebuild[simple-amd64-with-grub-uefi.xml]>
        <Function test_check_updates[simple-amd64-with-grub-uefi.xml]>

So either we need to fit building and validating extended image into this matrix (with a test_extended(), how I proposed above), or we need to take out simple-validation-image.xml out of it, and write its own special test for it, e.g. by tweaking test_base_extended_build().

The reason I want to keep the images in tests/base-extended/simple-validation/ (and the test for them) is that it's beneficial to test both 'rich' and 'minimal' images. Simple-validation-image.xml is used as a catch-all for all the features that need to be tested, and it will in the future gain even more, and be not so simple anymore. Those other images won't change much, and they're meant as a minimal example that's still bootable.

[t-8ch]

I would also be fine with removing simple-validation-image from the default test_simple_build().
Or we keep it separate, but then we still need some basic test for things like finetuning and archives.
If we split it out, we should also split out the test script though.

[kanavin]

Ok. I'll propose an alternative pull request that reuses the original simple validation image and extends the existing tests for it. Then we'll tweak from there.