From 3a69f90d598e27e9ed85eed367e412278606c4b3 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Mon, 12 Oct 2026 19:01:48 +0100 Subject: [PATCH 01/10] Check older supported image versions in the update-dependencies skill When following an image bump changes module source, the skill now asks for a check against the older versions the vendor still supports. Versions past end of life are out of scope. --- .agents/skills/update-dependencies/SKILL.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index aa1da46d3..04507a29d 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -27,6 +27,18 @@ Don't work from a fixed list of failure causes. Packages, images and registries The AGENTS.md rules on runtime dependencies and on module images apply to every bump. +## Older image versions + +Users choose their own image tag, so a module has to keep working with the older versions the vendor still supports, not only the one in its `Dockerfile`. Check this whenever following an image bump changes module source and not only tests: a new wait, endpoint, flag, default or log line, or a binary run inside the container. + +1. Find which versions are still supported, from the vendor's lifecycle page or endoflife.date. Versions past end of life are out of scope: don't test them, and don't add or keep code paths for them. +2. Find how far back the thing you now rely on exists, from the vendor's release notes or its source at older tags. +3. Run the changed path against the oldest supported version with a temporary local test, plus one in between if the gap is large. + - Check the tag with `docker manifest inspect ` first. A tag without an arm64 image runs emulated on Apple Silicon and may not start, so report it as not checked rather than guessing. + - Don't commit the test or add its images to the suite (AGENTS.md). +4. If a supported version can't take the change, keep the old path for it. Versions that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported version is a breaking change. +5. List in the PR body which versions were run and which were not. + ## An image no longer pulls - Find a replacement. Prefer, in order: From dcff735fba57c234350542d232e7696027290317 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Mon, 12 Oct 2026 19:21:22 +0100 Subject: [PATCH 02/10] Keep the oldest supported image version in version-sensitive module suites A check done by hand at bump time protects nothing that changes later, so modules whose source depends on what is inside the image now run their tests against the oldest supported version too. The update-dependencies skill covers adding that version on a bump and moving the pin at end of life. --- .agents/skills/update-dependencies/SKILL.md | 13 +++++++------ AGENTS.md | 4 ++++ 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index 04507a29d..76f4e8ab6 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -29,15 +29,16 @@ The AGENTS.md rules on runtime dependencies and on module images apply to every ## Older image versions -Users choose their own image tag, so a module has to keep working with the older versions the vendor still supports, not only the one in its `Dockerfile`. Check this whenever following an image bump changes module source and not only tests: a new wait, endpoint, flag, default or log line, or a binary run inside the container. +Users choose their own image tag, so a module has to keep working with the older versions the vendor still supports. Modules whose source depends on what is inside the image keep the oldest supported version in their suite (AGENTS.md "Tests"), because a check done by hand at bump time protects nothing that changes later. + +A bump that needs no change to module source can't break older versions: they still run the same code. When following a bump does change module source: 1. Find which versions are still supported, from the vendor's lifecycle page or endoflife.date. Versions past end of life are out of scope: don't test them, and don't add or keep code paths for them. -2. Find how far back the thing you now rely on exists, from the vendor's release notes or its source at older tags. -3. Run the changed path against the oldest supported version with a temporary local test, plus one in between if the gap is large. - - Check the tag with `docker manifest inspect ` first. A tag without an arm64 image runs emulated on Apple Silicon and may not start, so report it as not checked rather than guessing. - - Don't commit the test or add its images to the suite (AGENTS.md). +2. If the suite already runs against an older supported version, those tests passing is the check. +3. If it doesn't, add the oldest supported version to the suite in the same PR. Check the tag with `docker manifest inspect `. If it has no arm64 image, use the oldest supported tag that has both amd64 and arm64. 4. If a supported version can't take the change, keep the old path for it. Versions that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported version is a breaking change. -5. List in the PR body which versions were run and which were not. + +Dependabot does not bump a hardcoded tag. When working on a module's image bump, check its pinned older version too. Once that version is past end of life, move the pin to the oldest version still supported, or remove it if none is left. ## An image no longer pulls diff --git a/AGENTS.md b/AGENTS.md index 37dea63bf..14950c990 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,6 +57,10 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Reuse an image the tests already use, for example `cristianrgreco/testcontainer:1.1.14` in `packages/testcontainers`, or the module's own image. - Module tests read their images from `packages/modules//Dockerfile` (one `FROM` line per image) with `getImage(__dirname, index)`, so Dependabot can bump them. Do not hardcode images in module source or tests. - Exception: a second version of the *same* image must be hardcoded in its test file (for example `influxdb1-container.test.ts`, `kafka-container-7.test.ts`). Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. +- A module whose source depends on what is inside the image also runs its tests against the oldest version the vendor still supports, so a later change can't silently break that version. This is the one case where a suite pulls an extra image. + - It applies when the module runs a command in the container, or relies on an endpoint, flag, default or log line that differs between versions. + - Loop the tests over both images with `it.each` or `describe.each`, so new tests cover the older version by default. Skip it explicitly where a feature can't work there. A suite too heavy to double may loop a representative part of itself. + - Add the older version the first time a change adds or touches such a dependency. Versions past the vendor's end of life are out of scope. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). - Tests that rely on shared/global mocks (for example `vi.spyOn` on shared loggers/singletons) can be flaky due to interleaving or automatic mock resets. - Prefer asserting observable behavior instead of shared global mock state when possible. From 4e4318d73be13f6bb0bf49d12434627fdcb8749b Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Mon, 12 Oct 2026 19:26:34 +0100 Subject: [PATCH 03/10] Cover every supported version line in module tests Module tests now cover each version line the vendor still supports, not only the oldest one for version-sensitive modules: the whole suite on the newest and oldest lines, a smoke test for each line in between. The update-dependencies skill keeps the hardcoded lines current on each bump. --- .agents/skills/update-dependencies/SKILL.md | 19 +++++++++---------- AGENTS.md | 11 +++++++---- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index 76f4e8ab6..7d4aaae82 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -27,18 +27,17 @@ Don't work from a fixed list of failure causes. Packages, images and registries The AGENTS.md rules on runtime dependencies and on module images apply to every bump. -## Older image versions +## Supported image versions -Users choose their own image tag, so a module has to keep working with the older versions the vendor still supports. Modules whose source depends on what is inside the image keep the oldest supported version in their suite (AGENTS.md "Tests"), because a check done by hand at bump time protects nothing that changes later. +A module's tests cover every version line the vendor still supports (AGENTS.md "Tests"). Dependabot only bumps the `Dockerfile`, so the hardcoded lines are kept current here. On every image bump for a module: -A bump that needs no change to module source can't break older versions: they still run the same code. When following a bump does change module source: - -1. Find which versions are still supported, from the vendor's lifecycle page or endoflife.date. Versions past end of life are out of scope: don't test them, and don't add or keep code paths for them. -2. If the suite already runs against an older supported version, those tests passing is the check. -3. If it doesn't, add the oldest supported version to the suite in the same PR. Check the tag with `docker manifest inspect `. If it has no arm64 image, use the oldest supported tag that has both amd64 and arm64. -4. If a supported version can't take the change, keep the old path for it. Versions that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported version is a breaking change. - -Dependabot does not bump a hardcoded tag. When working on a module's image bump, check its pinned older version too. Once that version is past end of life, move the pin to the oldest version still supported, or remove it if none is left. +1. List the supported lines from the vendor's lifecycle page or endoflife.date. +2. Compare them with the lines the tests run: + - **Past end of life:** remove its tests. Removing module code that exists only for that line is a breaking change, so propose it separately. + - **Supported but untested:** add it as AGENTS.md "Tests" describes. + - **Hardcoded:** move it to that line's latest tag. `docker manifest inspect ` should list both amd64 and arm64. +3. When the bump moves the `Dockerfile` to a new line, hardcode the line it leaves in the tests in the same PR, if the vendor still supports it. +4. If a supported line fails with the module as changed for the bump, keep the old path for it. Lines that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported line is a breaking change. ## An image no longer pulls diff --git a/AGENTS.md b/AGENTS.md index 14950c990..6cad3f2ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,10 +57,13 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Reuse an image the tests already use, for example `cristianrgreco/testcontainer:1.1.14` in `packages/testcontainers`, or the module's own image. - Module tests read their images from `packages/modules//Dockerfile` (one `FROM` line per image) with `getImage(__dirname, index)`, so Dependabot can bump them. Do not hardcode images in module source or tests. - Exception: a second version of the *same* image must be hardcoded in its test file (for example `influxdb1-container.test.ts`, `kafka-container-7.test.ts`). Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. -- A module whose source depends on what is inside the image also runs its tests against the oldest version the vendor still supports, so a later change can't silently break that version. This is the one case where a suite pulls an extra image. - - It applies when the module runs a command in the container, or relies on an endpoint, flag, default or log line that differs between versions. - - Loop the tests over both images with `it.each` or `describe.each`, so new tests cover the older version by default. Skip it explicitly where a feature can't work there. A suite too heavy to double may loop a representative part of itself. - - Add the older version the first time a change adds or touches such a dependency. Versions past the vendor's end of life are out of scope. +- A module's tests cover every version line the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. + - The newest line comes from the module `Dockerfile`. Hardcode the others in the test file, each as that line's latest tag. + - Run the whole suite against the newest and the oldest supported line by looping it over both images with `describe.each` or `it.each`. New tests then cover the oldest line by default; skip it explicitly where a feature needs a newer version. + - Give each line in between one smoke test: start the container and do one round trip. + - A line that needs its own class, setup or assertions gets its own test file named after it, for example `kafka-container-7.test.ts`. + - A suite too heavy to double may loop a representative part of itself. Say which part in the PR. + - Versions past the vendor's end of life are out of scope: don't test them, and remove their tests. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). - Tests that rely on shared/global mocks (for example `vi.spyOn` on shared loggers/singletons) can be flaky due to interleaving or automatic mock resets. - Prefer asserting observable behavior instead of shared global mock state when possible. From fffebd0fcbfe91f111a31540e4f9e5097d4acb50 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 10:57:18 +0100 Subject: [PATCH 04/10] Describe the image list pattern and the removal of end-of-life code --- .agents/skills/update-dependencies/SKILL.md | 16 +++++++++------- AGENTS.md | 15 ++++++++------- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index 7d4aaae82..4a8ed8d15 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -29,15 +29,17 @@ The AGENTS.md rules on runtime dependencies and on module images apply to every ## Supported image versions -A module's tests cover every version line the vendor still supports (AGENTS.md "Tests"). Dependabot only bumps the `Dockerfile`, so the hardcoded lines are kept current here. On every image bump for a module: +A module's tests run against every version line the vendor still supports, from the `IMAGES` list in its test file (AGENTS.md "Tests"). Dependabot only bumps the `Dockerfile`, so the older lines in that list are kept current here. Do this on every image bump for a module, and when changing a module for any other reason: 1. List the supported lines from the vendor's lifecycle page or endoflife.date. -2. Compare them with the lines the tests run: - - **Past end of life:** remove its tests. Removing module code that exists only for that line is a breaking change, so propose it separately. - - **Supported but untested:** add it as AGENTS.md "Tests" describes. - - **Hardcoded:** move it to that line's latest tag. `docker manifest inspect ` should list both amd64 and arm64. -3. When the bump moves the `Dockerfile` to a new line, hardcode the line it leaves in the tests in the same PR, if the vendor still supports it. -4. If a supported line fails with the module as changed for the bump, keep the old path for it. Lines that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported line is a breaking change. +2. Compare them with the module's `IMAGES` list: + - **A listed line is past end of life:** remove it from the list. Remove the module code that exists only for it in a separate PR, since that is a breaking change. + - **A supported line is missing:** add the exact tag of its latest patch. + - **A listed line has a newer patch:** move it to that tag. +3. Check each new tag with `docker manifest inspect `. A registry can list a tag before its manifest exists, and the tag should have the same architectures as the `Dockerfile` image. +4. Check that the `Dockerfile` is on the newest supported line, and move it by hand if not. Dependabot has left a `Dockerfile` behind when the vendor changed tag format (Neo4j and ScyllaDB moved to calendar versions) and when the tag is not a plain version (SQL Server `2022-CU13-ubuntu-22.04`). +5. When the bump moves the `Dockerfile` to a new line, add the line it leaves to `IMAGES` in the same PR, if the vendor still supports it. +6. If a supported line fails with the module as changed for the bump, keep the old path for it. Lines that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported line is a breaking change. ## An image no longer pulls diff --git a/AGENTS.md b/AGENTS.md index 6cad3f2ca..94991ea8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -56,14 +56,15 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Every extra image increases how long the tests take to run and how much storage they need. - Reuse an image the tests already use, for example `cristianrgreco/testcontainer:1.1.14` in `packages/testcontainers`, or the module's own image. - Module tests read their images from `packages/modules//Dockerfile` (one `FROM` line per image) with `getImage(__dirname, index)`, so Dependabot can bump them. Do not hardcode images in module source or tests. - - Exception: a second version of the *same* image must be hardcoded in its test file (for example `influxdb1-container.test.ts`, `kafka-container-7.test.ts`). Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. + - Exception: a second version of the *same* image must be hardcoded in its test file (for example `influxdb1-container.test.ts`). Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. - A module's tests cover every version line the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. - - The newest line comes from the module `Dockerfile`. Hardcode the others in the test file, each as that line's latest tag. - - Run the whole suite against the newest and the oldest supported line by looping it over both images with `describe.each` or `it.each`. New tests then cover the oldest line by default; skip it explicitly where a feature needs a newer version. - - Give each line in between one smoke test: start the container and do one round trip. - - A line that needs its own class, setup or assertions gets its own test file named after it, for example `kafka-container-7.test.ts`. - - A suite too heavy to double may loop a representative part of itself. Say which part in the PR. - - Versions past the vendor's end of life are out of scope: don't test them, and remove their tests. + - Keep one `IMAGES` list in the test file: the older supported lines, oldest first, each as the exact tag of its latest patch, then the `Dockerfile` image. Don't use rolling tags such as `redis:7.4`. + - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every line. + - A line is supported while the vendor's free or community support covers it. Leave out a line that ends within 60 days, and a feature line that ends when the next release ships. If the vendor publishes no lifecycle, test only the `Dockerfile` image. + - Where a test can't pass on a line, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. + - A line that needs its own class or setup gets its own test file named after it, for example `influxdb1-container.test.ts`. + - A suite too heavy to run on every line may loop a representative part of itself. Say which part in the PR. + - Versions past the vendor's end of life are out of scope. Remove their tests and the module code that exists only for them. Removing that code is a breaking change, so it goes in its own PR. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). - Tests that rely on shared/global mocks (for example `vi.spyOn` on shared loggers/singletons) can be flaky due to interleaving or automatic mock resets. - Prefer asserting observable behavior instead of shared global mock state when possible. From 05db204e32b9f2895613d974482adf1e63650bec Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 11:07:42 +0100 Subject: [PATCH 05/10] Use a generic test file name and note when to run a looped suite sequentially --- AGENTS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 94991ea8e..a1e8829e0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -56,14 +56,15 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Every extra image increases how long the tests take to run and how much storage they need. - Reuse an image the tests already use, for example `cristianrgreco/testcontainer:1.1.14` in `packages/testcontainers`, or the module's own image. - Module tests read their images from `packages/modules//Dockerfile` (one `FROM` line per image) with `getImage(__dirname, index)`, so Dependabot can bump them. Do not hardcode images in module source or tests. - - Exception: a second version of the *same* image must be hardcoded in its test file (for example `influxdb1-container.test.ts`). Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. + - Exception: a second version of the *same* image must be hardcoded in its test file. Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. - A module's tests cover every version line the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. - Keep one `IMAGES` list in the test file: the older supported lines, oldest first, each as the exact tag of its latest patch, then the `Dockerfile` image. Don't use rolling tags such as `redis:7.4`. - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every line. - A line is supported while the vendor's free or community support covers it. Leave out a line that ends within 60 days, and a feature line that ends when the next release ships. If the vendor publishes no lifecycle, test only the `Dockerfile` image. - Where a test can't pass on a line, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. - - A line that needs its own class or setup gets its own test file named after it, for example `influxdb1-container.test.ts`. + - A line that needs its own class or setup gets its own test file named after it, such as `-container-v.test.ts`. - A suite too heavy to run on every line may loop a representative part of itself. Say which part in the PR. + - Looping multiplies the suite's container starts. If that overloads the test runner (tests far slower than locally, connection timeouts, passing only on retry), consider running the suite sequentially with `{ concurrent: false }` on its `describe`, and point the change out to the user. - Versions past the vendor's end of life are out of scope. Remove their tests and the module code that exists only for them. Removing that code is a breaking change, so it goes in its own PR. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). - Tests that rely on shared/global mocks (for example `vi.spyOn` on shared loggers/singletons) can be flaky due to interleaving or automatic mock resets. From 0196d2ded718696fd3c0460d5d44527c14a9cdae Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 11:12:46 +0100 Subject: [PATCH 06/10] Cover the latest release of each supported major, not every minor --- .agents/skills/update-dependencies/SKILL.md | 16 ++++++++-------- AGENTS.md | 14 +++++++------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index 4a8ed8d15..b9907593a 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -29,17 +29,17 @@ The AGENTS.md rules on runtime dependencies and on module images apply to every ## Supported image versions -A module's tests run against every version line the vendor still supports, from the `IMAGES` list in its test file (AGENTS.md "Tests"). Dependabot only bumps the `Dockerfile`, so the older lines in that list are kept current here. Do this on every image bump for a module, and when changing a module for any other reason: +A module's tests run against every major version the vendor still supports, from the `IMAGES` list in its test file (AGENTS.md "Tests"). Dependabot only bumps the `Dockerfile`, so the older majors in that list are kept current here. Do this on every image bump for a module, and when changing a module for any other reason: -1. List the supported lines from the vendor's lifecycle page or endoflife.date. +1. List the supported major versions from the vendor's lifecycle page or endoflife.date. 2. Compare them with the module's `IMAGES` list: - - **A listed line is past end of life:** remove it from the list. Remove the module code that exists only for it in a separate PR, since that is a breaking change. - - **A supported line is missing:** add the exact tag of its latest patch. - - **A listed line has a newer patch:** move it to that tag. + - **A listed major is past end of life:** remove it from the list. Remove the module code that exists only for it in a separate PR, since that is a breaking change. + - **A supported major is missing:** add the exact tag of its latest release. + - **A listed major has a newer minor or patch:** move it to that tag. 3. Check each new tag with `docker manifest inspect `. A registry can list a tag before its manifest exists, and the tag should have the same architectures as the `Dockerfile` image. -4. Check that the `Dockerfile` is on the newest supported line, and move it by hand if not. Dependabot has left a `Dockerfile` behind when the vendor changed tag format (Neo4j and ScyllaDB moved to calendar versions) and when the tag is not a plain version (SQL Server `2022-CU13-ubuntu-22.04`). -5. When the bump moves the `Dockerfile` to a new line, add the line it leaves to `IMAGES` in the same PR, if the vendor still supports it. -6. If a supported line fails with the module as changed for the bump, keep the old path for it. Lines that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported line is a breaking change. +4. Check that the `Dockerfile` is on the newest supported version, and move it by hand if not. Dependabot has left a `Dockerfile` behind when the vendor changed tag format (Neo4j and ScyllaDB moved to calendar versions) and when the tag is not a plain version (SQL Server `2022-CU13-ubuntu-22.04`). +5. When the bump moves the `Dockerfile` to a new major, add the major it leaves to `IMAGES` in the same PR, if the vendor still supports it. +6. If a supported major fails with the module as changed for the bump, keep the old path for it. Majors that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported major is a breaking change. ## An image no longer pulls diff --git a/AGENTS.md b/AGENTS.md index a1e8829e0..0927152d0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,13 +57,13 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Reuse an image the tests already use, for example `cristianrgreco/testcontainer:1.1.14` in `packages/testcontainers`, or the module's own image. - Module tests read their images from `packages/modules//Dockerfile` (one `FROM` line per image) with `getImage(__dirname, index)`, so Dependabot can bump them. Do not hardcode images in module source or tests. - Exception: a second version of the *same* image must be hardcoded in its test file. Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. -- A module's tests cover every version line the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. - - Keep one `IMAGES` list in the test file: the older supported lines, oldest first, each as the exact tag of its latest patch, then the `Dockerfile` image. Don't use rolling tags such as `redis:7.4`. - - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every line. - - A line is supported while the vendor's free or community support covers it. Leave out a line that ends within 60 days, and a feature line that ends when the next release ships. If the vendor publishes no lifecycle, test only the `Dockerfile` image. - - Where a test can't pass on a line, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. - - A line that needs its own class or setup gets its own test file named after it, such as `-container-v.test.ts`. - - A suite too heavy to run on every line may loop a representative part of itself. Say which part in the PR. +- A module's tests cover every major version the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. + - Keep one `IMAGES` list in the test file: the latest release of each older supported major, oldest first, as an exact tag, then the `Dockerfile` image. Don't list other minors or patches of a major, and don't use rolling tags such as `redis:7`. + - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every major. + - A major is supported while the vendor's free or community support covers it. Leave out a major that ends within 60 days. If the vendor publishes no lifecycle, test only the `Dockerfile` image. + - Where a test can't pass on a major, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. + - A major that needs its own class or setup gets its own test file named after it, such as `-container-v.test.ts`. + - A suite too heavy to run on every major may loop a representative part of itself. Say which part in the PR. - Looping multiplies the suite's container starts. If that overloads the test runner (tests far slower than locally, connection timeouts, passing only on retry), consider running the suite sequentially with `{ concurrent: false }` on its `describe`, and point the change out to the user. - Versions past the vendor's end of life are out of scope. Remove their tests and the module code that exists only for them. Removing that code is a breaking change, so it goes in its own PR. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). From f3edf245b3f5ad29620765d8c97cd6173dad99ef Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 12:45:06 +0100 Subject: [PATCH 07/10] Run looped suites one image at a time with concurrent tests --- .agents/skills/add-module/SKILL.md | 1 + AGENTS.md | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.agents/skills/add-module/SKILL.md b/.agents/skills/add-module/SKILL.md index 3322878bf..1d30586db 100644 --- a/.agents/skills/add-module/SKILL.md +++ b/.agents/skills/add-module/SKILL.md @@ -31,6 +31,7 @@ For the container class, tests and docs page, start from the module closest in s - Every test does a real client round trip, such as write then read, or publish then receive. Asserting that getters or connection strings look right proves nothing. Cover the default path and each option that changes behaviour. - Use `await using`, with one container per test. Tests run concurrently. - Import from the container file, not `./index`. Read images with `getImage(__dirname, index)` (AGENTS.md). +- If the vendor supports more than one major version of the image, loop the suite over an `IMAGES` list (AGENTS.md "Tests"). - Wrap the part a user would copy in `// name {` … `// }` markers inside the `it` body. The docs include these blocks, so give each marker a name that appears nowhere else in the file (AGENTS.md). ## Docs and finish diff --git a/AGENTS.md b/AGENTS.md index 0927152d0..da6757cc3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,11 +60,12 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - A module's tests cover every major version the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. - Keep one `IMAGES` list in the test file: the latest release of each older supported major, oldest first, as an exact tag, then the `Dockerfile` image. Don't list other minors or patches of a major, and don't use rolling tags such as `redis:7`. - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every major. + - Run the majors one after another and each major's tests concurrently, as a single-image suite runs: pass `{ concurrent: false }` to `describe.each` and declare the tests with `it.concurrent`. - A major is supported while the vendor's free or community support covers it. Leave out a major that ends within 60 days. If the vendor publishes no lifecycle, test only the `Dockerfile` image. - Where a test can't pass on a major, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. - A major that needs its own class or setup gets its own test file named after it, such as `-container-v.test.ts`. - A suite too heavy to run on every major may loop a representative part of itself. Say which part in the PR. - - Looping multiplies the suite's container starts. If that overloads the test runner (tests far slower than locally, connection timeouts, passing only on retry), consider running the suite sequentially with `{ concurrent: false }` on its `describe`, and point the change out to the user. + - If a looped suite still overloads the test runner (tests far slower than locally, connection timeouts, passing only on retry), consider running its tests sequentially too, with plain `it`, and point the change out to the user. - Versions past the vendor's end of life are out of scope. Remove their tests and the module code that exists only for them. Removing that code is a breaking change, so it goes in its own PR. - Vitest runs tests concurrently by default (`sequence.concurrent: true` in `vitest.config.ts`). - Tests that rely on shared/global mocks (for example `vi.spyOn` on shared loggers/singletons) can be flaky due to interleaving or automatic mock resets. From f4f6ef974a9d214026853087407cff569a3eb795 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 13:06:01 +0100 Subject: [PATCH 08/10] Say which images the supported-versions rule covers --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index da6757cc3..318be2bc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,6 +59,7 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Exception: a second version of the *same* image must be hardcoded in its test file. Dependabot treats same-image `FROM` lines as one dependency and bumps them all to the newest tag. - A module's tests cover every major version the vendor still supports, so a change can't silently break one that users still run. This is the one case where a suite pulls extra images. - Keep one `IMAGES` list in the test file: the latest release of each older supported major, oldest first, as an exact tag, then the `Dockerfile` image. Don't list other minors or patches of a major, and don't use rolling tags such as `redis:7`. + - This applies to every image the module itself runs, so a second class, edition or variant with its own image gets its own list. Images that only help the tests, such as a CLI or the database behind an emulator, are exempt. - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every major. - Run the majors one after another and each major's tests concurrently, as a single-image suite runs: pass `{ concurrent: false }` to `describe.each` and declare the tests with `it.concurrent`. - A major is supported while the vendor's free or community support covers it. Leave out a major that ends within 60 days. If the vendor publishes no lifecycle, test only the `Dockerfile` image. From 81f7f62db8b46bb3021aaae76d5e2135df8d69d6 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 16:47:04 +0100 Subject: [PATCH 09/10] Check images hardcoded in module source when bumping a module --- .agents/skills/update-dependencies/SKILL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/skills/update-dependencies/SKILL.md b/.agents/skills/update-dependencies/SKILL.md index b9907593a..bf66bf043 100644 --- a/.agents/skills/update-dependencies/SKILL.md +++ b/.agents/skills/update-dependencies/SKILL.md @@ -40,6 +40,7 @@ A module's tests run against every major version the vendor still supports, from 4. Check that the `Dockerfile` is on the newest supported version, and move it by hand if not. Dependabot has left a `Dockerfile` behind when the vendor changed tag format (Neo4j and ScyllaDB moved to calendar versions) and when the tag is not a plain version (SQL Server `2022-CU13-ubuntu-22.04`). 5. When the bump moves the `Dockerfile` to a new major, add the major it leaves to `IMAGES` in the same PR, if the vendor still supports it. 6. If a supported major fails with the module as changed for the bump, keep the old path for it. Majors that differ in ports, auth or startup get their own class (`add-module`). Dropping a supported major is a breaking change. +7. Check the module source for an image the module starts itself, such as a video recorder or the database behind an emulator. Dependabot doesn't see those. Move it to its latest exact tag and run the module's tests. ## An image no longer pulls From 167f28b56ca7dbce5cbec280bbecf4bc7a823f97 Mon Sep 17 00:00:00 2001 From: Cristian Greco Date: Tue, 13 Oct 2026 17:12:48 +0100 Subject: [PATCH 10/10] Describe selecting tests by image without a module-specific example --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 318be2bc5..5892f01b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,7 +63,7 @@ If a skill or this file turns out to be wrong or incomplete, update it in the sa - Loop the whole suite over the list with `describe.each(IMAGES)`, so every test runs on every major. - Run the majors one after another and each major's tests concurrently, as a single-image suite runs: pass `{ concurrent: false }` to `describe.each` and declare the tests with `it.concurrent`. - A major is supported while the vendor's free or community support covers it. Leave out a major that ends within 60 days. If the vendor publishes no lifecycle, test only the `Dockerfile` image. - - Where a test can't pass on a major, skip it by matching the image, for example `it.skipIf(image.startsWith("elasticsearch:8."))`, with a comment saying why. + - Where a test only applies to some of the images, select it by matching the image with `it.skipIf` or `it.runIf`, and add a comment when the reason isn't obvious. - A major that needs its own class or setup gets its own test file named after it, such as `-container-v.test.ts`. - A suite too heavy to run on every major may loop a representative part of itself. Say which part in the PR. - If a looped suite still overloads the test runner (tests far slower than locally, connection timeouts, passing only on retry), consider running its tests sequentially too, with plain `it`, and point the change out to the user.