From bc2c104944ae0da057e473a2441c593bdd66a482 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 01/18] ci: run a composition's ci-cleanup workflow after every
 mzcompose job

Cancelling or timing out a Buildkite job ends the mzcompose process with
SIGTERM, which skips Python `finally` blocks, so a composition that holds
resources outside of Docker never gets to release them. For
cluster-spec-sheet this meant a cancelled staging shard never ran
`mz region disable --hard`, and its region stayed up until the same E2E
account came around again in the 7-of-20 rotation, which can take weeks.
On 2026-09-03 two cancelled builds left nine staging regions behind; one
of them, at 20,000 materialized views, kept the staging eu-west-1
CockroachDB cluster at 75-85% CPU for over two days and stalled the
staging release rollout.

The mzcompose plugin now runs a composition's `ci-cleanup` workflow, if it
defines one, from its exit trap: after the main workflow has exited for
any reason, before the Docker teardown, with the main workflow's
arguments. A failure is written to run.log, where ci-annotate-errors now
matches it into the error annotation and fails an otherwise green job.

Two details keep the cleanup run from interfering with the main run. It
writes no JUnit report: it runs under the same BUILDKITE_JOB_ID, so its
report would overwrite the main run's, which for a normally failing job is
the only source of the failure annotation. And the hook kills the main
run's containers first: SIGTERM ends the mzcompose process, but a `docker
compose run` container it started keeps going, and an in-flight `mz region
enable` could otherwise re-create the region after the cleanup deleted it.

No composition defines the workflow yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/README.md                | 14 ++++++++++
 ci/plugins/mzcompose/hooks/command            | 26 +++++++++++++++++++
 .../materialize/cli/ci_annotate_errors.py     |  1 +
 misc/python/materialize/cli/mzcompose.py      | 12 +++++++--
 4 files changed, 51 insertions(+), 2 deletions(-)

diff --git a/ci/plugins/mzcompose/README.md b/ci/plugins/mzcompose/README.md
index b66ab43e9f5c8..c754f6e1dfea1 100644
--- a/ci/plugins/mzcompose/README.md
+++ b/ci/plugins/mzcompose/README.md
@@ -24,4 +24,18 @@ at the last occurrence of the marker. Logs without the marker are scanned in
 full. This is useful for workflows that exercise historical binaries before
 testing the current build.
 
+## Cleaning up resources outside of Docker
+
+A composition that creates resources outside of Docker, such as a Cloud
+region, can define a workflow named `ci-cleanup`. The command hook runs it
+after the main workflow has exited, however it exited, and passes it the same
+`args`. Cancelling or timing out a job ends the main workflow with SIGTERM,
+which does not run Python `finally` blocks, so a composition must not rely on
+its own cleanup path for those cases. Before the workflow runs, the hook kills
+the main workflow's containers, so nothing left over from the main run can
+race the cleanup; the Docker teardown proper happens afterwards. The workflow
+must be idempotent: it also runs after a successful run that already cleaned
+up. It writes no JUnit report, so the main workflow's report survives. Its
+failure is recorded in the error annotation and fails an otherwise green job.
+
 [Buildkite plugin]: https://buildkite.com/docs/agent/v3/plugins
diff --git a/ci/plugins/mzcompose/hooks/command b/ci/plugins/mzcompose/hooks/command
index a4fa58e6eecd7..d33fb6c2df6c3 100644
--- a/ci/plugins/mzcompose/hooks/command
+++ b/ci/plugins/mzcompose/hooks/command
@@ -145,6 +145,32 @@ cleanup() {
     printf "\n%s" "$BUILDKITE_LABEL: test timed out" >> run.log
   fi
 
+  # A composition that holds resources outside of Docker, such as a Cloud
+  # region, can declare a `ci-cleanup` workflow. It runs here, before the Docker
+  # teardown, with the arguments of the main workflow. A cancelled or timed-out
+  # job reaches this trap through SIGTERM, which ends the mzcompose process
+  # without running the composition's own cleanup path, so this is the only
+  # cleanup such a run gets. A failure goes to run.log, where ci-annotate-errors
+  # turns it into an error annotation and fails an otherwise green job.
+  if echo "$workflows" | grep -x "ci-cleanup" > /dev/null; then
+    ci_unimportant_heading ":docker: Running the composition's ci-cleanup workflow"
+    # The main workflow's containers can outlive its process: SIGTERM ends
+    # mzcompose, but a `docker compose run` container it started keeps going,
+    # and an in-flight `mz region enable` could re-create the region after the
+    # cleanup deleted it. Only compose-managed containers, so that containers
+    # a composition drives through other means, such as kind nodes, keep
+    # running for the log collection below. Killed containers keep their logs
+    # for services.log; `mzcompose down` removes them.
+    docker ps --quiet --filter label=com.docker.compose.project | xargs --no-run-if-empty docker kill > /dev/null || true
+    # 15m keeps a hung cleanup from eating the agents' 40-minute cancel grace
+    # period before the artifacts and the error annotation. Normal disables
+    # take about 30 s. The output also goes to run.log so that the artifact
+    # shows why a cleanup failed.
+    if ! bin/ci-builder run "$builder" timeout 15m bin/mzcompose --find "$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" run ci-cleanup "${run_args[@]:1}" |& sed -u -r 's/\x1B\[[0-9;]*[A-Za-z]//g' | tee -a run.log; then
+      printf "\n%s" "$BUILDKITE_LABEL: ci-cleanup workflow failed" >> run.log
+    fi
+  fi
+
   ci_unimportant_heading "Post command steps"
   # Run before potential "run down" in coverage
   docker ps --all --quiet | xargs --no-run-if-empty docker inspect | jq '
diff --git a/misc/python/materialize/cli/ci_annotate_errors.py b/misc/python/materialize/cli/ci_annotate_errors.py
index 6bed1881c5ad5..3097f3f2dbc2e 100644
--- a/misc/python/materialize/cli/ci_annotate_errors.py
+++ b/misc/python/materialize/cli/ci_annotate_errors.py
@@ -94,6 +94,7 @@
     | SUMMARY:\ .*Sanitizer
     | primary\ source\ \w+\ seemingly\ dropped\ before\ subsource
     | :\ test\ timed\ out
+    | :\ ci-cleanup\ workflow\ failed
     | very\ slow\ coordinator\ message
     # Only notifying on unexpected failures. INT, TRAP, BUS, FPE, SEGV, PIPE
     | \ ANOM_ABEND\ .*\ sig=(2|5|7|8|11|13)
diff --git a/misc/python/materialize/cli/mzcompose.py b/misc/python/materialize/cli/mzcompose.py
index c438e09ee9e05..b7b314ca74cfd 100644
--- a/misc/python/materialize/cli/mzcompose.py
+++ b/misc/python/materialize/cli/mzcompose.py
@@ -869,7 +869,7 @@ def handle_composition(
                     workflow_name, *args.unknown_subargs[1:], *extra_args
                 )
 
-            if self.shall_generate_junit_report(args.find, composition):
+            if self.shall_generate_junit_report(args.find, workflow_name, composition):
                 junit_suite = self.generate_junit_suite(composition)
                 self.write_junit_report_to_file(junit_suite)
 
@@ -880,8 +880,16 @@ def handle_composition(
                 raise UIError("at least one test case failed")
 
     def shall_generate_junit_report(
-        self, composition_name: str | None, composition: Composition
+        self,
+        composition_name: str | None,
+        workflow_name: str | None,
+        composition: Composition,
     ) -> bool:
+        if workflow_name == "ci-cleanup":
+            # Runs from the CI plugin's exit trap after the main workflow, under
+            # the same BUILDKITE_JOB_ID, so its report would overwrite the main
+            # run's. Its outcome is reported through run.log instead.
+            return False
         if composition.has_testdrive_junit:
             # Testdrive already produced a junit.xml with detailed errors;
             # skip the mzcompose-level junit to avoid duplicate annotations.

From f9a784f219f67fead0b97aa69704e7179b7ed3e4 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 02/18] cluster-spec-sheet: extract target construction from
 workflow_default

Pure move: the `--target` argument and the per-target construction of the
bench target and the `mz` service go into `add_target_arguments` and
`make_target`, so that another workflow can build the same target from the
same arguments. No behavior change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 87 ++++++++++++++++------------
 1 file changed, 49 insertions(+), 38 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 0cdcbdab910c2..c7ead5cbb023a 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3748,16 +3748,60 @@ def log_environment_info(target: "BenchTarget") -> None:
             pass
 
 
-def workflow_default(composition: Composition, parser: WorkflowArgumentParser) -> None:
-    """
-    Run the bench workflow by default
-    """
+def add_target_arguments(parser: argparse.ArgumentParser) -> None:
+    """The arguments of a workflow that constructs a bench target."""
     parser.add_argument(
         "--cleanup",
         default=False,
         action=argparse.BooleanOptionalAction,
         help="Destroy the region at the end of the workflow.",
     )
+    parser.add_argument(
+        "--target",
+        default="cloud-production",
+        choices=["cloud-production", "cloud-staging", "docker"],
+        help="Target to deploy to (default: cloud-production).",
+    )
+
+
+def make_target(composition: Composition, target: str) -> tuple["BenchTarget", Mz]:
+    """The bench target for `--target`, with the `mz` service configured for it."""
+    if target == "cloud-production":
+        bench_target: BenchTarget = CloudTarget(
+            composition, PRODUCTION_USERNAME, PRODUCTION_APP_PASSWORD or ""
+        )
+        mz = Mz(
+            region=PRODUCTION_REGION,
+            environment=PRODUCTION_ENVIRONMENT,
+            app_password=PRODUCTION_APP_PASSWORD or "",
+        )
+    elif target == "cloud-staging":
+        staging_username, staging_app_password = staging_credentials()
+        bench_target = CloudTarget(
+            composition,
+            staging_username,
+            staging_app_password,
+            is_staging=True,
+            version=staging_version(),
+        )
+        mz = Mz(
+            region=STAGING_REGION,
+            environment=STAGING_ENVIRONMENT,
+            app_password=staging_app_password,
+        )
+    elif target == "docker":
+        bench_target = DockerTarget(composition)
+        mz = Mz(app_password="")
+    else:
+        raise ValueError(f"Unknown target: {target}")
+    return bench_target, mz
+
+
+def workflow_default(composition: Composition, parser: WorkflowArgumentParser) -> None:
+    """
+    Run the bench workflow by default
+    """
+    add_target_arguments(parser)
     parser.add_argument(
         "--record",
         default=f"results_{int(time.time())}.csv",
@@ -3769,12 +3813,6 @@ def workflow_default(composition: Composition, parser: WorkflowArgumentParser) -
         action=argparse.BooleanOptionalAction,
         help="Analyze results after completing test. Dispatches to cluster-scale or envd-scale focused analyses based on the file suffix: `.cluster.csv` or `.envd.csv`.",
     )
-    parser.add_argument(
-        "--target",
-        default="cloud-production",
-        choices=["cloud-production", "cloud-staging", "docker"],
-        help="Target to deploy to (default: cloud-production).",
-    )
     parser.add_argument(
         "--max-scale",
         type=int,
@@ -3865,34 +3903,7 @@ def workflow_default(composition: Composition, parser: WorkflowArgumentParser) -
             f"{', '.join(sorted(cluster_object_limits_requested))}."
         )
 
-    if args.target == "cloud-production":
-        target: BenchTarget = CloudTarget(
-            composition, PRODUCTION_USERNAME, PRODUCTION_APP_PASSWORD or ""
-        )
-        mz = Mz(
-            region=PRODUCTION_REGION,
-            environment=PRODUCTION_ENVIRONMENT,
-            app_password=PRODUCTION_APP_PASSWORD or "",
-        )
-    elif args.target == "cloud-staging":
-        staging_username, staging_app_password = staging_credentials()
-        target: BenchTarget = CloudTarget(
-            composition,
-            staging_username,
-            staging_app_password,
-            is_staging=True,
-            version=staging_version(),
-        )
-        mz = Mz(
-            region=STAGING_REGION,
-            environment=STAGING_ENVIRONMENT,
-            app_password=staging_app_password,
-        )
-    elif args.target == "docker":
-        target = DockerTarget(composition)
-        mz = Mz(app_password="")
-    else:
-        raise ValueError(f"Unknown target: {args.target}")
+    target, mz = make_target(composition, args.target)
 
     with composition.override(mz):
         target_max = target.max_scale()

From 8fe835b9ec24b2939f07e736f3f159142d9cebc1 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 03/18] cluster-spec-sheet: define the ci-cleanup workflow and
 require --target

`ci-cleanup` destroys the region of a Cloud target started with
`--cleanup` and does nothing otherwise. The CI mzcompose plugin runs it
after `default` has exited, however it exited, with the same arguments, so
a cancelled or timed-out staging shard now disables its region instead of
leaving it up. It also runs after a successful run, where the region is
already gone; `mz region disable` reports an absent region as success.

The workflow runs unattended and reads the main workflow's arguments with
`parse_known_args`, so `--target` is now required instead of defaulting to
cloud-production: a mistyped target used to fall through to production
with cleanup enabled. Every CI step passes the target explicitly; ad-hoc
local runs get an error instead of production.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/README.md    | 18 +++++++++++----
 test/cluster-spec-sheet/mzcompose.py | 34 +++++++++++++++++++++++++---
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/test/cluster-spec-sheet/README.md b/test/cluster-spec-sheet/README.md
index 83ba8edc435a5..b29d95ddc6e2b 100644
--- a/test/cluster-spec-sheet/README.md
+++ b/test/cluster-spec-sheet/README.md
@@ -4,9 +4,12 @@ Reproduce data for the cluster spec sheet effort.
 
 # Usage
 
-`bin/mzcompose --find cluster-spec-sheet run default`
+`bin/mzcompose --find cluster-spec-sheet run default --target=<target>`
 
 This will run all scenarios currently defined for the cluster spec sheet.
+`--target` is required: `cloud-production`, `cloud-staging`, or `docker`.
+There is deliberately no default, because the CI cleanup below destroys the
+target's region unattended.
 
 Pass `--cleanup` to disable the region after the test.
 
@@ -15,7 +18,12 @@ Pass `--cleanup` to disable the region after the test.
 
 ## Running via Buildkite
 
-The workload runs as part of the release qualification pipeline in Buildkite.
+The workload runs in the `spec-sheet` Buildkite pipeline. After a CI job
+ends, however it ends, the mzcompose plugin runs the `ci-cleanup` workflow
+with the job's arguments; for a Cloud target started with `--cleanup` it
+disables the region. A canceled or timed-out job never reaches the
+composition's own cleanup, so this is what keeps canceled runs from leaving
+regions behind.
 
 ## Running manually in Cloud
 
@@ -27,7 +35,7 @@ Once the environment variables have been set, you can run:
 
 ```
 cd test/cluster-spec-sheet
-./mzcompose run default
+./mzcompose run default --target=cloud-production
 ```
 
 ## Running in Docker
@@ -55,11 +63,11 @@ bin/mzcompose --find cluster-spec-sheet run default envd_qps_scalability  --targ
 ```
 or
 ```
-bin/mzcompose --find cluster-spec-sheet run default cluster
+bin/mzcompose --find cluster-spec-sheet run default cluster --target=cloud-production
 ```
 or
 ```
-bin/mzcompose --find cluster-spec-sheet run default envd_objects_scalability
+bin/mzcompose --find cluster-spec-sheet run default envd_objects_scalability --target=cloud-production
 ```
 or
 ```
diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index c7ead5cbb023a..3227f483958b9 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3749,18 +3749,22 @@ def log_environment_info(target: "BenchTarget") -> None:
 
 
 def add_target_arguments(parser: argparse.ArgumentParser) -> None:
-    """The arguments of a workflow that constructs a bench target."""
+    """The arguments `workflow_default` and `workflow_ci_cleanup` share."""
     parser.add_argument(
         "--cleanup",
         default=False,
         action=argparse.BooleanOptionalAction,
         help="Destroy the region at the end of the workflow.",
     )
+    # Required rather than defaulting to cloud-production: `ci-cleanup` runs
+    # unattended from the CI plugin's exit trap and destroys the target's
+    # region, so a missing or malformed target must fail loudly instead of
+    # quietly selecting production. Every CI step passes it explicitly.
     parser.add_argument(
         "--target",
-        default="cloud-production",
+        required=True,
         choices=["cloud-production", "cloud-staging", "docker"],
-        help="Target to deploy to (default: cloud-production).",
+        help="Target to deploy to.",
     )
 
 
@@ -3797,6 +3801,30 @@ def make_target(composition: Composition, target: str) -> tuple["BenchTarget", M
     return bench_target, mz
 
 
+def workflow_ci_cleanup(
+    composition: Composition, parser: WorkflowArgumentParser
+) -> None:
+    """
+    Destroy the Cloud region of a run that did not get to its own cleanup.
+
+    The CI mzcompose plugin runs this workflow after `default` has exited, however
+    it exited, with the same arguments. A cancelled or timed-out job ends `default`
+    with SIGTERM, which skips its `finally` block, so for such a run this is the
+    only region cleanup there is. Only `--cleanup` and `--target` are read.
+    """
+    add_target_arguments(parser)
+    args, _ = parser.parse_known_args()
+    if not args.cleanup:
+        print("Not destroying the region: the run was started without --cleanup")
+        return
+    target, mz = make_target(composition, args.target)
+    if not isinstance(target, CloudTarget):
+        print(f"Nothing to clean up for --target={args.target}")
+        return
+    with composition.override(mz):
+        target.cleanup()
+
+
 def workflow_default(composition: Composition, parser: WorkflowArgumentParser) -> None:
     """
     Run the bench workflow by default

From 876db8eb8dc8d1b05f33597700bc7add6bc86591 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 04/18] cluster-spec-sheet: verify the region is gone after
 cleanup, stop swallowing errors

`disable_region` caught every `UIError` to tolerate a 404, but `mz region
disable` already reports an absent region as success, so the catch only hid
real failures (auth, API errors) behind a green job. Remove it.

`CloudTarget.cleanup` now also runs `mz region list` and fails unless the
target's region is reported disabled. Returning from the disable is not
proof, and a surviving region keeps costing money and CockroachDB load until
someone notices. The check runs after every cleanup, including the unattended
one from the CI plugin's exit trap, whose failure lands in the error
annotation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 48 +++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 8 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 3227f483958b9..1e8bbe38c8782 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -15,6 +15,7 @@
 import csv
 import glob
 import itertools
+import json
 import os
 import re
 import shlex
@@ -3386,16 +3387,40 @@ def teardown(self, runner: ScenarioRunner) -> None:
 # TODO: We should factor the region helpers below out into a separate module.
 # (Similar `disable_region` functions also occur in other tests.)
 def disable_region(composition: Composition, hard: bool) -> None:
+    # `mz region disable` treats an already-disabled region as success, so any
+    # failure here is a real one (auth, API error) and must surface.
     print("Shutting down region ...")
+    if hard:
+        composition.run("mz", "region", "disable", "--hard", rm=True)
+    else:
+        composition.run("mz", "region", "disable", rm=True)
+
+
+def verify_region_disabled(composition: Composition, region: str) -> None:
+    """Fail unless `mz region list` reports `region` as disabled.
 
+    `mz region disable` returning is not proof: it can be answered by a stale
+    or failing API, and a region that survives a cleanup keeps costing money
+    and load until someone notices. "disabled" also covers a region whose
+    deletion is still pending, so the check proves that no enabled region is
+    left, not that the deletion has completed. The check runs after every
+    cleanup, including the unattended one from the CI plugin's exit trap.
+    """
+    output = composition.run(
+        "mz", "region", "list", "--format", "json", rm=True, capture_and_print=True
+    ).stdout
+    # The JSON array is the last line: `mz region list` reports a cloud provider
+    # whose lookup failed as an `Error: ...` line on stdout before it.
     try:
-        if hard:
-            composition.run("mz", "region", "disable", "--hard", rm=True)
-        else:
-            composition.run("mz", "region", "disable", rm=True)
-    except UIError:
-        # Can return: status 404 Not Found
-        pass
+        regions = json.loads(output.strip().splitlines()[-1])
+    except (ValueError, IndexError) as e:
+        raise UIError(f"unexpected `mz region list` output: {output!r}") from e
+    for entry in regions:
+        if entry["region"] == region:
+            if entry["status"] == "disabled":
+                return
+            raise UIError(f"region {region} is still {entry['status']} after disable")
+    raise UIError(f"region {region} missing from `mz region list` output")
 
 
 def enable_region(target: "CloudTarget", envd_cpus: int | None = None) -> None:
@@ -3772,7 +3797,10 @@ def make_target(composition: Composition, target: str) -> tuple["BenchTarget", M
     """The bench target for `--target`, with the `mz` service configured for it."""
     if target == "cloud-production":
         bench_target: BenchTarget = CloudTarget(
-            composition, PRODUCTION_USERNAME, PRODUCTION_APP_PASSWORD or ""
+            composition,
+            PRODUCTION_USERNAME,
+            PRODUCTION_APP_PASSWORD or "",
+            region=PRODUCTION_REGION,
         )
         mz = Mz(
             region=PRODUCTION_REGION,
@@ -3785,6 +3813,7 @@ def make_target(composition: Composition, target: str) -> tuple["BenchTarget", M
             composition,
             staging_username,
             staging_app_password,
+            region=STAGING_REGION,
             is_staging=True,
             version=staging_version(),
         )
@@ -4055,11 +4084,13 @@ def __init__(
         composition: Composition,
         username: str,
         app_password: str,
+        region: str,
         is_staging: bool = False,
         version: str | None = None,
     ) -> None:
         self.composition = composition
         self.username = username
+        self.region = region
         self.app_password = app_password
         self.new_app_password: str | None = None
         self.is_staging = is_staging
@@ -4122,6 +4153,7 @@ def new_connection(self) -> psycopg.Connection:
 
     def cleanup(self) -> None:
         disable_region(self.composition, hard=True)
+        verify_region_disabled(self.composition, self.region)
 
     # M.1 size with the same worker count as the {scale}00cc size. Scales
     # above 8 have no available M.1 equivalent.

From 806679769b668b319c686811f8d07fcae97686d4 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 05/18] cluster-spec-sheet: correct the staging credentials
 paragraph in the README

The README named `NIGHTLY_CANARY_USERNAME` and `NIGHTLY_CANARY_APP_PASSWORD`
for staging runs, which the composition does not read. A staging run uses
one account from the E2E database pool, selected by
`CI_CONCURRENCY_POOL_SLOT` with the matching
`E2E_STAGING_TEST_FRONTEGG_DATABASE_APP_PASSWORD_<n>`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/cluster-spec-sheet/README.md b/test/cluster-spec-sheet/README.md
index b29d95ddc6e2b..1af34a72d6c0c 100644
--- a/test/cluster-spec-sheet/README.md
+++ b/test/cluster-spec-sheet/README.md
@@ -27,7 +27,7 @@ regions behind.
 
 ## Running manually in Cloud
 
-To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, you need to set the environment variables `NIGHTLY_CANARY_USERNAME` and `NIGHTLY_CANARY_APP_PASSWORD`.
+To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, the run uses one account from the E2E database pool: set `E2E_STAGING_TEST_FRONTEGG_DATABASE_APP_PASSWORD_<n>` for the pool index `<n>` and select it with `CI_CONCURRENCY_POOL_SLOT=<n>` (outside CI, index 0 is used when the slot is unset); the username is derived from the index.
 
 The username is an email address, the app password is a password generated in the cloud console (something like `mzp_...`).
 

From 517377be15f888b591f211bcb1244b3c8f649d14 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 16:44:21 +0200
Subject: [PATCH 06/18] ci: label the staging spec-sheet step by what it runs

The two production steps read "<what runs> (against <where>)", while the
staging step said only where it runs. It runs the envd QPS and envd
objects scalability groups and the cluster object limits group, so its
label now follows the same pattern, with "object count" spelled out so
the limit is not mistaken for a size limit. The "Cluster spec sheet"
prefix stays, which is what the mzcompose plugin's services.log check
matches on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/spec-sheet/pipeline.template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/spec-sheet/pipeline.template.yml b/ci/spec-sheet/pipeline.template.yml
index a9dac5052a1c4..0550b40081816 100644
--- a/ci/spec-sheet/pipeline.template.yml
+++ b/ci/spec-sheet/pipeline.template.yml
@@ -111,7 +111,7 @@ steps:
       queue: linux-aarch64-small
 
   - id: cluster-spec-sheet-staging
-    label: "Cluster spec sheet: Staging"
+    label: "Cluster spec sheet: envd Scalability + Cluster Object Count Limits (against Staging)"
     timeout_in_minutes: 3600
     depends_on: devel-docker-tags
     parallelism: 7

From afd85f6665ad78b20c2656e862b90917fcaa4ded Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:28 +0200
Subject: [PATCH 07/18] cluster-spec-sheet: retry the sweep's disable together
 with its enable

cloud_recreate_region_with_envd_cpus retries a failed `mz region enable`
because the staging Cloud API returns the occasional 502 during a sweep.
With disable_region no longer swallowing errors, a 502 on the disable
would have aborted the sweep instead. Start each attempt inside the
retry, so the disable-before-retry invariant holds and the protection
covers both calls; the warning names the step generically since either
call can fail. The comment no longer claims that the CLI does not retry
the enable: both `mz region enable` and `mz region disable` retry their
API call for up to 12 minutes, and the 502s that reach this loop are
the ones that survive that.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 1e8bbe38c8782..6f4c984b5167b 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3505,19 +3505,19 @@ def cloud_recreate_region_with_envd_cpus(
     soon as it has booted, in about a minute.
     """
     for attempt in range(1, attempts + 1):
-        disable_region(target.composition, hard=True)
         try:
+            disable_region(target.composition, hard=True)
             enable_region(target, envd_cpus=envd_cpus)
             break
         except UIError as e:
             # A sweep does a dozen of these calls and the staging Cloud API returns the
-            # occasional 502, which `mz region enable` does not retry on its own. We
-            # disable again before retrying, so that a half-finished enable can't leave
-            # the region running with the previous allocation.
+            # occasional 502 that surfaces despite the CLI's own retries. A retry starts
+            # over with the disable, so that a half-finished enable can't leave the
+            # region running with the previous allocation.
             if attempt == attempts:
                 raise
             print(
-                f"WARNING: 'mz region enable' failed (attempt {attempt}/{attempts}): {e}"
+                f"WARNING: recreating the region failed (attempt {attempt}/{attempts}): {e}"
             )
             time.sleep(30)
 

From 52fa425910b57bd81c9f3271df8b2c875daf4f19 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:28 +0200
Subject: [PATCH 08/18] cluster-spec-sheet: say where `mz region disable`
 handles a missing region

The comment claimed the CLI treats an already-disabled region as success
without saying how, while two other compositions still guard the same
call against a 404. Point at the CLI's mapping so the contract can be
checked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 6f4c984b5167b..ebc79d55167a2 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3387,8 +3387,9 @@ def teardown(self, runner: ScenarioRunner) -> None:
 # TODO: We should factor the region helpers below out into a separate module.
 # (Similar `disable_region` functions also occur in other tests.)
 def disable_region(composition: Composition, hard: bool) -> None:
-    # `mz region disable` treats an already-disabled region as success, so any
-    # failure here is a real one (auth, API error) and must surface.
+    # `mz region disable` reports a region that does not exist as success (its
+    # `disable` maps the API's 404 to "Region already disabled"), so any failure
+    # here is a real one (auth, API error) and must surface.
     print("Shutting down region ...")
     if hard:
         composition.run("mz", "region", "disable", "--hard", rm=True)

From 6e0375463898dfd4f2e00f304b97f5bff2f149c6 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:28 +0200
Subject: [PATCH 09/18] ci: kill only the composition's own containers before
 ci-cleanup

The pre-cleanup kill matched every compose-managed container on the
host. Scope it to the compose project named after the job's composition,
which is what the comment described, and drop the assumption that a host
runs one job at a time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/hooks/command | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/ci/plugins/mzcompose/hooks/command b/ci/plugins/mzcompose/hooks/command
index d33fb6c2df6c3..7c5be25ef1eec 100644
--- a/ci/plugins/mzcompose/hooks/command
+++ b/ci/plugins/mzcompose/hooks/command
@@ -157,11 +157,12 @@ cleanup() {
     # The main workflow's containers can outlive its process: SIGTERM ends
     # mzcompose, but a `docker compose run` container it started keeps going,
     # and an in-flight `mz region enable` could re-create the region after the
-    # cleanup deleted it. Only compose-managed containers, so that containers
-    # a composition drives through other means, such as kind nodes, keep
-    # running for the log collection below. Killed containers keep their logs
-    # for services.log; `mzcompose down` removes them.
-    docker ps --quiet --filter label=com.docker.compose.project | xargs --no-run-if-empty docker kill > /dev/null || true
+    # cleanup deleted it. Only this composition's compose project (named after
+    # the composition): other jobs' containers, and containers a composition
+    # drives outside compose, such as kind nodes, keep running for the log
+    # collection below. Killed containers keep their logs for services.log;
+    # `mzcompose down` removes them.
+    docker ps --quiet --filter "label=com.docker.compose.project=$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" | xargs --no-run-if-empty docker kill > /dev/null || true
     # 15m keeps a hung cleanup from eating the agents' 40-minute cancel grace
     # period before the artifacts and the error annotation. Normal disables
     # take about 30 s. The output also goes to run.log so that the artifact

From 890744ee5c4736ce087eebc7191f19f5ace9574d Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:28 +0200
Subject: [PATCH 10/18] ci: hard-kill a ci-cleanup that ignores its timeout

`timeout 15m` only sends SIGTERM, so a cleanup that ignores it would
escape the bound the comment promises. Add `--kill-after=30s`, as
run_mzcompose does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/hooks/command | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/plugins/mzcompose/hooks/command b/ci/plugins/mzcompose/hooks/command
index 7c5be25ef1eec..63e04b635510e 100644
--- a/ci/plugins/mzcompose/hooks/command
+++ b/ci/plugins/mzcompose/hooks/command
@@ -167,7 +167,7 @@ cleanup() {
     # period before the artifacts and the error annotation. Normal disables
     # take about 30 s. The output also goes to run.log so that the artifact
     # shows why a cleanup failed.
-    if ! bin/ci-builder run "$builder" timeout 15m bin/mzcompose --find "$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" run ci-cleanup "${run_args[@]:1}" |& sed -u -r 's/\x1B\[[0-9;]*[A-Za-z]//g' | tee -a run.log; then
+    if ! bin/ci-builder run "$builder" timeout --signal=TERM --kill-after=30s 15m bin/mzcompose --find "$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" run ci-cleanup "${run_args[@]:1}" |& sed -u -r 's/\x1B\[[0-9;]*[A-Za-z]//g' | tee -a run.log; then
       printf "\n%s" "$BUILDKITE_LABEL: ci-cleanup workflow failed" >> run.log
     fi
   fi

From 158bb3a5f50e180c6a9fade1f3686caf4f888114 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:28 +0200
Subject: [PATCH 11/18] cluster-spec-sheet: fail with a UIError on any
 unexpected `mz region list` output

A changed field name raised KeyError instead of UIError, and a
pretty-printed array would not have parsed from its last line. Parse
from the first line that starts the array to the end, and turn every
parse and shape surprise into the same UIError with the raw output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 30 +++++++++++++++++-----------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index ebc79d55167a2..7cc86afbf0677 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3404,24 +3404,30 @@ def verify_region_disabled(composition: Composition, region: str) -> None:
     or failing API, and a region that survives a cleanup keeps costing money
     and load until someone notices. "disabled" also covers a region whose
     deletion is still pending, so the check proves that no enabled region is
-    left, not that the deletion has completed. The check runs after every
-    cleanup, including the unattended one from the CI plugin's exit trap.
+    left, not that the deletion has completed.
     """
     output = composition.run(
         "mz", "region", "list", "--format", "json", rm=True, capture_and_print=True
     ).stdout
-    # The JSON array is the last line: `mz region list` reports a cloud provider
-    # whose lookup failed as an `Error: ...` line on stdout before it.
+    # `mz region list` reports a cloud provider whose lookup failed as an
+    # `Error: ...` line on stdout before the JSON array, so parsing the whole
+    # output would fail on an unrelated provider's hiccup.
+    lines = output.strip().splitlines()
+    start = next((i for i, line in enumerate(lines) if line.startswith("[")), None)
     try:
-        regions = json.loads(output.strip().splitlines()[-1])
-    except (ValueError, IndexError) as e:
+        if start is None:
+            raise ValueError("no JSON array in the output")
+        statuses = {
+            entry["region"]: entry["status"]
+            for entry in json.loads("\n".join(lines[start:]))
+        }
+    except (ValueError, KeyError, TypeError) as e:
         raise UIError(f"unexpected `mz region list` output: {output!r}") from e
-    for entry in regions:
-        if entry["region"] == region:
-            if entry["status"] == "disabled":
-                return
-            raise UIError(f"region {region} is still {entry['status']} after disable")
-    raise UIError(f"region {region} missing from `mz region list` output")
+    status = statuses.get(region)
+    if status is None:
+        raise UIError(f"region {region} missing from `mz region list` output")
+    if status != "disabled":
+        raise UIError(f"region {region} is still {status} after disable")
 
 
 def enable_region(target: "CloudTarget", envd_cpus: int | None = None) -> None:

From fcf9e0f780dfe6666bd7c383b9df3091245561c9 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:37:29 +0200
Subject: [PATCH 12/18] cluster-spec-sheet: pin the staging image version at
 the enable call site

make_target read BUILDKITE_COMMIT eagerly for cloud-staging, so a local
`run ci-cleanup --target=cloud-staging --cleanup` died with a KeyError
before disabling anything. Only `mz region enable` needs the version:
pass it there, inside the branch that already holds the other
staging-only flag, and drop the field that carried it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 7cc86afbf0677..8028d9f9bb301 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3449,12 +3449,13 @@ def enable_region(target: "CloudTarget", envd_cpus: int | None = None) -> None:
         # Production Cloud forbids callers from injecting environmentd args and rejects
         # `--environmentd-extra-arg` with a 403 Forbidden, so this is staging-only.
         args += [
-            "--environmentd-extra-arg=--system-parameter-default=with_0dt_caught_up_check_stability_period=0s"
+            "--environmentd-extra-arg=--system-parameter-default=with_0dt_caught_up_check_stability_period=0s",
+            # Pin the image built for this PR. Production does not accept a
+            # custom version.
+            "--version",
+            staging_version(),
         ]
 
-    if target.version is not None:
-        args += ["--version", target.version]
-
     target.composition.run("mz", "region", "enable", *args, rm=True)
 
 
@@ -3822,7 +3823,6 @@ def make_target(composition: Composition, target: str) -> tuple["BenchTarget", M
             staging_app_password,
             region=STAGING_REGION,
             is_staging=True,
-            version=staging_version(),
         )
         mz = Mz(
             region=STAGING_REGION,
@@ -4093,7 +4093,6 @@ def __init__(
         app_password: str,
         region: str,
         is_staging: bool = False,
-        version: str | None = None,
     ) -> None:
         self.composition = composition
         self.username = username
@@ -4101,11 +4100,6 @@ def __init__(
         self.app_password = app_password
         self.new_app_password: str | None = None
         self.is_staging = is_staging
-        # Set for staging runs so `mz region enable --version <version>` pins the
-        # exact image built for this PR. Must be None for production (production
-        # doesn't accept a custom version).
-        self.version = version
-        assert (version is not None) == is_staging
 
     def dbbench_connection_flags(self) -> list[str]:
         assert self.new_app_password is not None

From 827327a0e9837408ad1e33d3a3c6e9d52fb9efb3 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Sun, 6 Sep 2026 18:19:18 +0200
Subject: [PATCH 13/18] cluster-spec-sheet: upload the results before tearing
 the region down

The end-of-run region teardown sat in the `finally` ahead of the result
upload, the CSV artifact upload and the analysis. Now that the teardown
can fail (disable_region no longer swallows errors and the verify raises
on a region that is not reported disabled), such a failure at the end of
a multi-hour sweep would have discarded all of the run's data. Run the
uploads inside the outer `try` and tear the region down in its `finally`,
so a teardown failure still fails the run but no longer costs its
results. The CI plugin's ci-cleanup repeats the teardown afterwards.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 52 ++++++++++++++++------------
 1 file changed, 30 insertions(+), 22 deletions(-)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index 8028d9f9bb301..c676bd605d21e 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -4016,32 +4016,40 @@ def process(scenario_name: str) -> None:
 
         test_failed = True
         try:
-            scenarios_list = buildkite.shard_list(sorted(list(scenarios)), lambda s: s)
-            composition.test_parts(scenarios_list, process)
-            test_failed = False
-        finally:
+            try:
+                scenarios_list = buildkite.shard_list(
+                    sorted(list(scenarios)), lambda s: s
+                )
+                composition.test_parts(scenarios_list, process)
+                test_failed = False
+            finally:
+                for stream in streams.values():
+                    stream.file.close()
+
+            # Upload, archive, and analyze each result stream uniformly. The
+            # cluster_object_limits stream's extra `healthy` / `failure_mode`
+            # columns are silently dropped on upload (CSV writer uses
+            # extrasaction="ignore"); to recover them, consult the artifact
+            # CSV directly.
             for stream in streams.values():
-                stream.file.close()
-            if args.cleanup:
-                target.cleanup()
+                stream.spec.upload(composition, stream.path, not test_failed)
 
-        # Upload, archive, and analyze each result stream uniformly. The
-        # cluster_object_limits stream's extra `healthy` / `failure_mode`
-        # columns are silently dropped on upload (CSV writer uses
-        # extrasaction="ignore"); to recover them, consult the artifact
-        # CSV directly.
-        for stream in streams.values():
-            stream.spec.upload(composition, stream.path, not test_failed)
+            assert not test_failed
 
-        assert not test_failed
+            if buildkite.is_in_buildkite():
+                for stream in streams.values():
+                    buildkite.upload_artifact(stream.path, cwd=MZ_ROOT, quiet=True)
 
-        if buildkite.is_in_buildkite():
-            for stream in streams.values():
-                buildkite.upload_artifact(stream.path, cwd=MZ_ROOT, quiet=True)
-
-        if args.analyze:
-            for stream in streams.values():
-                stream.spec.analyze(stream.path)
+            if args.analyze:
+                for stream in streams.values():
+                    stream.spec.analyze(stream.path)
+        finally:
+            # Only after the results are out: the teardown can fail (a slow
+            # hard delete, a transient API error in its verify) and must not
+            # cost a multi-hour run its data. The CI plugin's `ci-cleanup`
+            # repeats the teardown after the workflow, however it ended.
+            if args.cleanup:
+                target.cleanup()
 
 
 class BenchTarget:

From 272d95b2a4835f3b42ade48efc88deb341df6693 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Mon, 7 Sep 2026 16:40:09 +0200
Subject: [PATCH 14/18] ci: end the run.log marker lines with a newline

The cleanup's output is appended to run.log right after the timed-out
marker, which had no trailing newline, so on a timed-out job the marker
line ran into the cleanup's first line. ci-annotate-errors still matched
it, but the annotation text carried the cleanup's output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/hooks/command | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ci/plugins/mzcompose/hooks/command b/ci/plugins/mzcompose/hooks/command
index 63e04b635510e..20ea6bdc3b062 100644
--- a/ci/plugins/mzcompose/hooks/command
+++ b/ci/plugins/mzcompose/hooks/command
@@ -142,7 +142,7 @@ cleanup() {
   END_TIME=$(date +%s)
   ELAPSED=$((END_TIME - START_TIME))
   if [ $ELAPSED -ge $((BUILDKITE_TIMEOUT * 60)) ]; then
-    printf "\n%s" "$BUILDKITE_LABEL: test timed out" >> run.log
+    printf "\n%s\n" "$BUILDKITE_LABEL: test timed out" >> run.log
   fi
 
   # A composition that holds resources outside of Docker, such as a Cloud
@@ -168,7 +168,7 @@ cleanup() {
     # take about 30 s. The output also goes to run.log so that the artifact
     # shows why a cleanup failed.
     if ! bin/ci-builder run "$builder" timeout --signal=TERM --kill-after=30s 15m bin/mzcompose --find "$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" run ci-cleanup "${run_args[@]:1}" |& sed -u -r 's/\x1B\[[0-9;]*[A-Za-z]//g' | tee -a run.log; then
-      printf "\n%s" "$BUILDKITE_LABEL: ci-cleanup workflow failed" >> run.log
+      printf "\n%s\n" "$BUILDKITE_LABEL: ci-cleanup workflow failed" >> run.log
     fi
   fi
 

From 9397e713a071a56cfc1640aea470fcece698de56 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Mon, 7 Sep 2026 16:52:53 +0200
Subject: [PATCH 15/18] ci: skip the pre-cleanup container kill under coverage

The coverage teardown stops containers gracefully on purpose, while
`docker kill` is SIGKILL. Killing first would defeat that for any
composition that adopts ci-cleanup and runs under coverage; none does
today.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/hooks/command | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ci/plugins/mzcompose/hooks/command b/ci/plugins/mzcompose/hooks/command
index 20ea6bdc3b062..540e67184457e 100644
--- a/ci/plugins/mzcompose/hooks/command
+++ b/ci/plugins/mzcompose/hooks/command
@@ -161,8 +161,12 @@ cleanup() {
     # the composition): other jobs' containers, and containers a composition
     # drives outside compose, such as kind nodes, keep running for the log
     # collection below. Killed containers keep their logs for services.log;
-    # `mzcompose down` removes them.
-    docker ps --quiet --filter "label=com.docker.compose.project=$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" | xargs --no-run-if-empty docker kill > /dev/null || true
+    # `mzcompose down` removes them. Skipped under coverage: the coverage
+    # teardown below stops containers gracefully on purpose, and a kill here
+    # would defeat it.
+    if [ -z "${CI_COVERAGE_ENABLED:-}" ]; then
+      docker ps --quiet --filter "label=com.docker.compose.project=$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" | xargs --no-run-if-empty docker kill > /dev/null || true
+    fi
     # 15m keeps a hung cleanup from eating the agents' 40-minute cancel grace
     # period before the artifacts and the error annotation. Normal disables
     # take about 30 s. The output also goes to run.log so that the artifact

From a9d7fa5bcd1c79bb4e78c8c199716bad3993cad0 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Mon, 7 Sep 2026 16:52:53 +0200
Subject: [PATCH 16/18] ci: spell out the ci-cleanup contract, and let the lint
 accept the workflow

The README now says what an adopter cannot infer from the hook: the
workflow gets the main workflow's full argument list and must use
parse_known_args, the pre-cleanup kill is SIGKILL and skipped under
coverage, and the cleanup has 15 minutes. check-mzcompose-files no
longer counts workflow_ci_cleanup, since its remedy for a composition
with two workflows, looping over c.workflows from default, would run the
cleanup in the middle of the test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 ci/plugins/mzcompose/README.md                | 31 +++++++++++++------
 .../lint-main/checks/check-mzcompose-files.sh |  3 +-
 2 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/ci/plugins/mzcompose/README.md b/ci/plugins/mzcompose/README.md
index c754f6e1dfea1..486c02ea5a995 100644
--- a/ci/plugins/mzcompose/README.md
+++ b/ci/plugins/mzcompose/README.md
@@ -28,14 +28,27 @@ testing the current build.
 
 A composition that creates resources outside of Docker, such as a Cloud
 region, can define a workflow named `ci-cleanup`. The command hook runs it
-after the main workflow has exited, however it exited, and passes it the same
-`args`. Cancelling or timing out a job ends the main workflow with SIGTERM,
-which does not run Python `finally` blocks, so a composition must not rely on
-its own cleanup path for those cases. Before the workflow runs, the hook kills
-the main workflow's containers, so nothing left over from the main run can
-race the cleanup; the Docker teardown proper happens afterwards. The workflow
-must be idempotent: it also runs after a successful run that already cleaned
-up. It writes no JUnit report, so the main workflow's report survives. Its
-failure is recorded in the error annotation and fails an otherwise green job.
+after the main workflow has exited, however it exited, and passes it the main
+workflow's full argument list: the step's `args` plus any `CI_EXTRA_ARGS`. The
+workflow must therefore parse with `parse_known_args` and find its target from
+those arguments alone. Cancelling or timing out a job ends the main workflow
+with SIGTERM, which does not run Python `finally` blocks, so a composition
+must not rely on its own cleanup path for those cases.
+
+Before the workflow runs, the hook kills (SIGKILL) the containers of the main
+run's compose project, so that a command still in flight from the main run,
+such as an `mz region enable` that outlived the cancelled process, cannot undo
+the cleanup after it has finished; the Docker teardown proper happens
+afterwards. The kill is skipped under `CI_COVERAGE_ENABLED`, where the Docker
+teardown is deliberately graceful. The workflow must be idempotent: it also
+runs after a successful run that already cleaned up. It has 15 minutes, so
+that a hung cleanup cannot eat the cancel grace period before the artifacts
+are uploaded. It writes no JUnit report, so the main workflow's report
+survives. Its failure is recorded in the error annotation and fails an
+otherwise green job.
+
+The mzcompose-files lint does not count `ci-cleanup` as a workflow, so a
+composition with a single main workflow can add it without looping over
+`c.workflows`; a `default` that does loop must skip `ci-cleanup`.
 
 [Buildkite plugin]: https://buildkite.com/docs/agent/v3/plugins
diff --git a/ci/test/lint-main/checks/check-mzcompose-files.sh b/ci/test/lint-main/checks/check-mzcompose-files.sh
index 9045e3d8934c2..9de0f9661eecd 100755
--- a/ci/test/lint-main/checks/check-mzcompose-files.sh
+++ b/ci/test/lint-main/checks/check-mzcompose-files.sh
@@ -59,7 +59,8 @@ check_default_workflow_references_others() {
     )
 
     for file in "${MZCOMPOSE_TEST_FILES[@]}"; do
-      MATCHES_COUNT=$(grep "def workflow_" "$file" -c)
+      # `ci-cleanup` is run by the mzcompose plugin's hook, never by `default`.
+      MATCHES_COUNT=$(grep "def workflow_" "$file" | grep -vc "def workflow_ci_cleanup" || true)
 
       if (( MATCHES_COUNT > 1 )); then
         # mzcompose file contains more than one workflow

From aaf18966ba117c5f2a27a769fbdeed233c0340a8 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Mon, 7 Sep 2026 16:52:54 +0200
Subject: [PATCH 17/18] cluster-spec-sheet: compute the staging version once
 per run

staging_version() runs `cargo metadata` and is now called on every
`mz region enable`, including inside the QPS sweep's retry loop, which
catches only UIError. Cache it, so a cargo failure cannot escape that
loop and the metadata is read once.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/mzcompose.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index c676bd605d21e..af1c103d283fb 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -13,6 +13,7 @@
 
 import argparse
 import csv
+import functools
 import glob
 import itertools
 import json
@@ -105,7 +106,10 @@ def staging_credentials() -> tuple[str, str]:
 }
 
 
+@functools.cache
 def staging_version() -> str:
+    # Cached: the version cannot change within a run, and `parse_cargo` shells
+    # out to cargo, whose failure must not surface inside a region retry loop.
     return f"{MzVersion.parse_cargo()}--pr.g{os.environ['BUILDKITE_COMMIT']}"
 
 

From 4623ba10da7f60ecd26151d51fc2b8b80a4ca436 Mon Sep 17 00:00:00 2001
From: Gabor Gevay <gabor@materialize.com>
Date: Mon, 7 Sep 2026 16:52:54 +0200
Subject: [PATCH 18/18] cluster-spec-sheet: document the CI half of --cleanup
 and the staging commit requirement

`--cleanup` also drives the CI plugin's ci-cleanup after a cancel or
timeout, and a manual staging run needs BUILDKITE_COMMIT for the image
version it enables.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
 test/cluster-spec-sheet/README.md    | 2 +-
 test/cluster-spec-sheet/mzcompose.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/cluster-spec-sheet/README.md b/test/cluster-spec-sheet/README.md
index 1af34a72d6c0c..6328c5b57726f 100644
--- a/test/cluster-spec-sheet/README.md
+++ b/test/cluster-spec-sheet/README.md
@@ -27,7 +27,7 @@ regions behind.
 
 ## Running manually in Cloud
 
-To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, the run uses one account from the E2E database pool: set `E2E_STAGING_TEST_FRONTEGG_DATABASE_APP_PASSWORD_<n>` for the pool index `<n>` and select it with `CI_CONCURRENCY_POOL_SLOT=<n>` (outside CI, index 0 is used when the slot is unset); the username is derived from the index.
+To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, the run uses one account from the E2E database pool: set `E2E_STAGING_TEST_FRONTEGG_DATABASE_APP_PASSWORD_<n>` for the pool index `<n>` and select it with `CI_CONCURRENCY_POOL_SLOT=<n>` (outside CI, index 0 is used when the slot is unset); the username is derived from the index. Staging runs also need `BUILDKITE_COMMIT`, which selects the image version to enable.
 
 The username is an email address, the app password is a password generated in the cloud console (something like `mzp_...`).
 
diff --git a/test/cluster-spec-sheet/mzcompose.py b/test/cluster-spec-sheet/mzcompose.py
index af1c103d283fb..8192475a73be0 100644
--- a/test/cluster-spec-sheet/mzcompose.py
+++ b/test/cluster-spec-sheet/mzcompose.py
@@ -3791,7 +3791,7 @@ def add_target_arguments(parser: argparse.ArgumentParser) -> None:
         "--cleanup",
         default=False,
         action=argparse.BooleanOptionalAction,
-        help="Destroy the region at the end of the workflow.",
+        help="Destroy the region when the workflow ends, and from CI's ci-cleanup after a cancel or timeout.",
     )
     # Required rather than defaulting to cloud-production: `ci-cleanup` runs
     # unattended from the CI plugin's exit trap and destroys the target's