Skip to content

Document testing modules against every supported image version - #1587

Merged
cristianrgreco merged 11 commits into
mainfrom
claude/check-older-images-in-update-dependencies
Oct 13, 2026
Merged

cristianrgreco merged 11 commits into
mainfrom
claude/check-older-images-in-update-dependencies

Conversation

@cristianrgreco

@cristianrgreco cristianrgreco commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Users choose their own image tag, so a module has to keep working with every version the vendor still supports. Nothing in the guidance covered that, and a check done by hand when an image is bumped protects nothing that changes later, such as a new feature that only works on the newest version.

This makes the supported versions a permanent part of each module's suite:

  • AGENTS.md "Tests": a module's tests cover every major version the vendor still supports.
    • Each test file keeps one IMAGES list: the latest release of each older supported major, oldest first, as an exact tag, then the Dockerfile image. Other minors and patches of a major are not listed. The older majors are hardcoded because Dependabot bumps every same-image FROM line to the newest tag.
    • The rule covers every image the module itself runs (a second class, edition or variant gets its own list), not images that only help the tests.
    • The whole suite loops over the list with describe.each(IMAGES), so every test runs on every major, including tests added later.
    • The majors run one after another and each major's tests run concurrently, as a single-image suite runs.
    • A major counts while the vendor's free or community support covers it. A major ending within 60 days is left out; with no published lifecycle, only the Dockerfile image is tested.
    • A test that only applies to some of the images is selected by matching the image.
    • A major that needs its own class or setup gets its own test file named after it.
    • A suite too heavy to run on every major may loop a representative part of itself.
    • If a looped suite still overloads the test runner, its tests may run sequentially too, and the change is pointed out to the user.
    • Versions past the vendor's end of life are out of scope: their tests are removed, and so is the module code that exists only for them, in its own PR because that is a breaking change.
  • add-module skill: one line pointing at the rule for modules whose vendor supports more than one major.
  • update-dependencies skill: a new "Supported image versions" section. On every image bump, and when a module changes for another reason, it compares the vendor's supported majors with the module's IMAGES list: drop majors past end of life, add missing supported majors, move listed majors to their latest release, verify new tags with docker manifest inspect, check that the Dockerfile is on the newest version, add the major the Dockerfile leaves when it moves to a new one, and check images the module starts itself from a tag in its source, which Dependabot doesn't see (Update the Selenium video recorder and Service Bus SQL Server images #1600 updates the two that exist).

This came out of #1514, where a module change started relying on a binary inside the image (mcstat) and compatibility with Couchbase 7.6 had to be checked by hand.

No module is changed here. The existing modules are brought in line in #1589, #1591, #1592, #1593, #1595 and #1596, and the code for end-of-life versions is removed in #1597. The wording was revised after that work to match what it settled on.

Verification

  • npm run format, npm run lint, npm run knip: pass.
  • npm run lint prints one info diagnostic that is already on main: biome.json references the 2.5.14 schema while the installed Biome is 2.5.15.

Not breaking

Guidance for agents and contributors only. No package source, tests or published files change.

When following an image bump changes module source, the skill now asks for
a check against the older versions the vendor still supports. Versions past
end of life are out of scope.
@cristianrgreco cristianrgreco added maintenance Improvements that do not change functionality patch Backward compatible bug fix labels Oct 12, 2026
…uites

A check done by hand at bump time protects nothing that changes later, so
modules whose source depends on what is inside the image now run their tests
against the oldest supported version too. The update-dependencies skill
covers adding that version on a bump and moving the pin at end of life.
@cristianrgreco cristianrgreco changed the title Check older supported image versions in the update-dependencies skill Document testing modules against the oldest supported image version Oct 12, 2026
Module tests now cover each version line the vendor still supports, not only
the oldest one for version-sensitive modules: the whole suite on the newest
and oldest lines, a smoke test for each line in between. The
update-dependencies skill keeps the hardcoded lines current on each bump.
@cristianrgreco
cristianrgreco merged commit b2f9c04 into main Oct 13, 2026
15 checks passed
@cristianrgreco
cristianrgreco deleted the claude/check-older-images-in-update-dependencies branch October 13, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Improvements that do not change functionality patch Backward compatible bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant