Skip to content

check-kolla-images: resolve the deployed tags - #2833

Merged
berendt merged 11 commits into
mainfrom
kolla-tag-resolution
Oct 15, 2026
Merged

check-kolla-images: resolve the deployed tags#2833
berendt merged 11 commits into
mainfrom
kolla-tag-resolution

Conversation

@ideaship

@ideaship ideaship commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

OSISM 11.0.0 shipped ten kolla containers that request image tags its build never pushed (osism/issues#1462). The ten tags were fixed in 11.0.1. The release checks were added and tightened during the same days:

One gap stays open after all of these. No check works out the tag a deployment actually requests and compares it with what the build pushed:

  • The drift checks compare variable names on latest, where every kolla image carries the same series tag. They cannot see an image that inherits its parent's tag: prometheus_valkey_exporter_tag: "{{ prometheus_tag }}" is a valid reference, it just names a tag that only exists for Prometheus. The same is true of a producer-map value that names an image the series no longer builds (letsencrypt_webserver).
  • check-versions.py checks which versions latest/ pins, not the tags a release deploys.
  • The registry checks (check-kolla-images.py, check-osism-images.py) prove that every pushed image is complete and belongs to the release, not that anything deploys it under the tag it was pushed with.

The next release could ship the same class of defect again: a new nested image with an inherited tag, or a renamed image upstream.

This PR adds that check to check-kolla-images.py. After the registry checks, it resolves every kolla-ansible image reference (<x>_image_full) as Ansible would, per role, through these layers:

  • the role defaults of kolla-ansible, with OSISM's patches applied;
  • the versions.yml and the overlay of the kolla-ansible image;
  • osism/defaults, as the inventory-reconciler image ships it.

It then compares each resolved tag with the tags the build pushed, according to its SBOM:

  • A reference to a tag that was never pushed fails the check, even for a service that isn't enabled by default.
  • A pushed image that no container uses is listed, without failing the check.
  • For every mismatch, the output shows the chain of variables the tag was resolved through.

How it works

  • src/ansible_templar.py: a small model of ansible-core 2.18 templating for these variables, covering layered precedence, lazy and recursive resolution, ansible_eval_concat, and cycle detection. The check refuses a release that pins a different ansible-core version.
  • src/kolla_tags.py: resolves the references and compares them with the build. The result is a Report of mismatches, unresolvable references, and images no container uses.
  • src/kolla_tag_inputs.py: gathers the inputs in one of two modes:
    • predicted, while osism/kolla-ansible:<version> doesn't exist yet: the inputs are rebuilt from the sources the next build will use (upstream stable/<series>, container-image-kolla-ansible at its tag or main, and the osism/defaults release that base.yml pins);
    • recorded, once the image exists: the inputs are read from the commits and the SBOM digest recorded in the labels of the kolla-ansible and inventory-reconciler images. This is the label baseline that Record in every release image what it was built from issues#1463 introduced. Images built before that baseline get an error that says to use --predicted.
  • New options: --predicted, --kolla-ansible-version, --inventory-reconciler-version, --kolla-ansible-ref, --defaults-ref, and --base-dir/--remote-fallback.
  • Docs: the README section "Checking the kolla images of a tag" and the header of scripts/check-kolla-images.sh describe the check.

The drift checks kolla_version_chain_* stay: they catch a subset of these problems earlier, on latest. A static check of the producer-map values (every value in SBOM_IMAGE_TO_VERSION must name an image the release builds) would report the letsencrypt_webserver case weeks before a release too. It is not part of this PR; this check only catches its effect, at release time.

Built on the image labels of osism/issues#1463

Recorded mode is possible because every release image now records what it was built from. It reads these labels:

Without these labels, the inputs of a past release could only be recovered by downloading and unpacking the images, several hundred MB per run. Images built before the labels existed (11.0.1 and older) get an error that names --predicted instead of a fallback that downloads the images.

Testing

  • Unit tests cover the templating model, the resolution and comparison, and both input modes, using local git fixtures and a fake registry.
  • Regression fixture (tests/fixtures/kolla-tags-11.0.0.json.gz), with the inputs of the published 11.0.0:
  • Live runs:
    • --predicted v0.20261008.0 (11.0.0): Tags: 120 ok, 8 failed, 4 not deployed (10 of 134 image references).
    • --predicted v0.20261011.0 (11.0.1): Tags: 128 ok, 0 failed, 4 not deployed (0 of 134 image references).
    • v0.20261008.0 without --predicted: stops with the error for images built before the label baseline.
  • The full suite passes; flake8 and black are clean.

Related

🤖 Generated with Claude Code

ideaship added 11 commits October 14, 2026 07:21
@ideaship
Add Ansible 2.18 templating for image tags
164052e 
kolla-ansible sets the image tags of the kolla services in Ansible
variables, and some tags are templates over other variables. Reading
them correctly means rendering them the way ansible-core 2.18 does, so
src/ansible_templar.py implements that subset of Jinja2 templating for
the check that compares those tags with the released images.

ansible-core 2.18 renders with jinja2_native off. A template renders to
text, and only text that looks like a dict, a list or True/False is
converted back. A string without template markers is not templated at
all, so a tag such as "2026.10" keeps its text, while "{{ false }}"
used as a condition is a boolean.

Variables come from layers with a rank. A higher rank wins. Layers of
the same rank must agree on a value once each is resolved. Jinja2
looks up every name of a template before it evaluates the template,
including names in a branch it does not take. A variable whose
resolution fails (a recursive loop, a rendering error, a disagreement)
therefore becomes a failed value that raises only when it is used, so
an unused broken branch does not fail the reference.

requirements.txt gains Jinja2==3.1.6.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Compare kolla image references with a build
9d9df91 
check-kolla-images deploys containers from the *_image_full references
kolla-ansible declares per role. Nothing so far checked that the tag a
role deploys is the tag the release build pushed, so a role can keep
a stale tag, e.g. a sidecar inheriting its parent's tag, unnoticed.

Add kolla_tags.check, which resolves every role's *_image_full in the
role's scope with the Ansible 2.18 templar from ansible_templar: the
role defaults first, then the OSISM layers (overlay, versions.yml and
osism/defaults all/*.yml), which win. The image name is the last path
segment of the resolved reference and the tag what follows its last
colon. A reference naming an image the build pushed is compared with
the pushed tag; a difference is a mismatch. A reference that does not
resolve, or names no tag, is unresolvable. One naming an image the
build did not push is not a finding.

The check reports the images that have a mismatch or an unresolvable
reference, and the hops that led to each resolved value, so a reader
can see which layer set the tag.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Gather predicted kolla tag inputs
04fa8fa 
The kolla tag check compares the image references of kolla-ansible
with the tags of the kolla build. To resolve a reference, it needs
the inputs the kolla-ansible image is deployed from. Before the image
exists (step 1 of the release), those inputs have to be reconstructed
from the source repositories.

Add src/kolla_tag_inputs.py with gather(), which reads them in
predicted mode at the commits they are built from:

- role defaults: upstream kolla-ansible at the series' ref, with the
  patches of container-image-kolla-ansible applied as the image build
  applies them (patch -p1, forward, dry run first)
- versions.yml: container-image-kolla-ansible's template rendered with
  the same context as files/src/render-versions.py (the context keys
  are checked against that file in the tests), at the v<version> tag
  or main
- overlay: overlays/<series>/kolla-ansible.yml of the same commit
- defaults: osism/defaults all/*.yml at the defaults_version of the
  inventory-reconciler release, or --defaults-ref
- SBOM: the kolla release SBOM of the version, or of the build named
  in next/kolla-ansible-v<version>.yml on main

Versions are read from the tags of osism/release, and ansible-core
other than 2.18 is refused, as the templating of ansible-core 2.19
differs. The Registry wrapper reads the labels of osism/<image>:<tag>
by digest, from the amd64 image of an index.

tests/fixtures/render-versions.py.txt is a copy of
files/src/render-versions.py from container-image-kolla-ansible, so
the test can check the rendering context against it. It is not
linted, hence the suffix.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Read recorded kolla tag inputs from labels
1ab4ae5 
When osism/kolla-ansible:<version> exists, the tag check now reads the
inputs that image was built from, from its labels, instead of
predicting them: the release commit, the kolla-ansible commit, the
container-image-kolla-ansible commit, the SBOM digest and the defaults
commit of the inventory-reconciler image. Predicted mode remains for
images that do not exist yet or with --predicted.

A missing or unusable required label is an error that names the label
and points to --predicted for images built before the label baseline
(osism/issues#1463). A label naming a commit that is not in the
repository is an error as well, and so is an SBOM digest that Harbor
no longer serves. An SBOM of another kolla build is noted, as in
predicted mode. An inventory-reconciler image that does not exist yet
falls back to the predicted defaults with a note.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
check-kolla-images: resolve the deployed tags
549685c 
Add a tag resolution section to the check. For every container
kolla-ansible deploys, the image reference is resolved as Ansible
would and its tag is compared with the tag the build pushed. The
inputs come from kolla_tag_inputs, either reconstructed from the
sources (predicted) or read from the kolla-ansible image of the
release (recorded).

New options select the kolla-ansible and inventory-reconciler
versions, the refs of the source repositories, predicted mode and
local clones. Every mismatch or unresolvable reference fails the
check; images kolla-ansible does not deploy are only listed.

The script now requires Python 3.10 and jinja2.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Test the tag check against OSISM 11.0.0
2fab2da 
Add a regression test that runs the kolla image tag check against the
published OSISM 11.0.0 inputs: the SBOM image list, the role defaults,
versions.yml and overlay of osism/kolla-ansible:0.20261008.0, and the
osism/defaults files from osism/inventory-reconciler:0.20261008.0.

The fixture is tests/fixtures/kolla-tags-11.0.0.json.gz. It was read
from the images once, outside this repository, because those images
predate the label baseline that the check reads (osism/issues#1463).
The fixture is a snapshot and is not regenerated by the test.

11.0.0 shipped ten containers that request tags its build never pushed
(osism/issues#1462). The test asserts that the check reports exactly
those ten mismatches, that no reference is unresolvable, and the
reference and unconsumed-image counts for the release.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Document the tag check of check-kolla-images
c912ad2 
check-kolla-images.py now also checks that every container
kolla-ansible deploys requests a tag the build pushed. Describe the
check in the README section on checking the kolla images of a tag:
what it compares, the predicted and recorded input modes, and the
options that select other sources. Extend the header and usage of
check-kolla-images.sh with the new options.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Refuse a release with a per-release overlay
a375d75 
The kolla-ansible image build moves
overlays/release/<VERSION>/release-kolla-ansible.yml into the image
and the inventory-reconciler installs it as group_vars. The tag check
does not model that layer and would check against the wrong
variables, so it now stops with an error when the file exists in
either the v-prefixed or the plain spelling, in predicted and in
recorded mode.

The module header no longer claims an image-content fallback in
recorded mode; an image without labels is an error pointing to
--predicted. The tests also assert the mode in two recorded cases.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Catch more errors in the tag check
c3b4671 
check_tags let jinja2.TemplateError, OSError and a malformed SBOM
escape as a traceback. Run the check inside the try block so these end
the section with an Error line, after the sources have been gathered.

ansible_concat chained the already consumed head back onto its
argument even for a list, repeating the first two items. As in
ansible-core, only do so for a generator.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Update the tag check docs and wrapper header
687c669 
List the tag resolution options in the wrapper header, name Jinja2
among its dependencies, and note in the README that the printed
variable chain includes default() fallbacks that were read but not
needed.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
Test the sort of defaults files
a384f47 
Pass the defaults files out of name order so the sort in
osism_layers is exercised.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship marked this pull request as ready for review October 15, 2026 06:07
@ideaship
ideaship requested a review from berendt October 15, 2026 06:08
@ideaship ideaship self-assigned this Oct 15, 2026
@berendt
berendt merged commit 57a5756 into main Oct 15, 2026
3 checks passed
@berendt
berendt deleted the kolla-tag-resolution branch October 15, 2026 08:52
@github-project-automation github-project-automation Bot moved this from New to Done in Human Board Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@berendt berendt Awaiting requested review from berendt

Assignees

@ideaship ideaship

Labels

None yet

Projects

Human Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ideaship @berendt @osism-agent