ci: run a composition's ci-cleanup workflow after every mzcompose job

ci/plugins/mzcompose/README.md

@@ -24,4 +24,31 @@ at the last occurrence of the marker. Logs without the marker are scanned in
 full. This is useful for workflows that exercise historical binaries before
 testing the current build.
 
+## Cleaning up resources outside of Docker
+
+A composition that creates resources outside of Docker, such as a Cloud
+region, can define a workflow named `ci-cleanup`. The command hook runs it
+after the main workflow has exited, however it exited, and passes it the main
+workflow's full argument list: the step's `args` plus any `CI_EXTRA_ARGS`. The
+workflow must therefore parse with `parse_known_args` and find its target from
+those arguments alone. Cancelling or timing out a job ends the main workflow
+with SIGTERM, which does not run Python `finally` blocks, so a composition
+must not rely on its own cleanup path for those cases.
+
+Before the workflow runs, the hook kills (SIGKILL) the containers of the main
+run's compose project, so that a command still in flight from the main run,
+such as an `mz region enable` that outlived the cancelled process, cannot undo
+the cleanup after it has finished; the Docker teardown proper happens
+afterwards. The kill is skipped under `CI_COVERAGE_ENABLED`, where the Docker
+teardown is deliberately graceful. The workflow must be idempotent: it also
+runs after a successful run that already cleaned up. It has 15 minutes, so
+that a hung cleanup cannot eat the cancel grace period before the artifacts
+are uploaded. It writes no JUnit report, so the main workflow's report
+survives. Its failure is recorded in the error annotation and fails an
+otherwise green job.
+
+The mzcompose-files lint does not count `ci-cleanup` as a workflow, so a
+composition with a single main workflow can add it without looping over
+`c.workflows`; a `default` that does loop must skip `ci-cleanup`.
+
 [Buildkite plugin]: https://buildkite.com/docs/agent/v3/plugins

ci/plugins/mzcompose/hooks/command

@@ -142,7 +142,38 @@ cleanup() {
   END_TIME=$(date +%s)
   ELAPSED=$((END_TIME - START_TIME))
   if [ $ELAPSED -ge $((BUILDKITE_TIMEOUT * 60)) ]; then
-    printf "\n%s" "$BUILDKITE_LABEL: test timed out" >> run.log
+    printf "\n%s\n" "$BUILDKITE_LABEL: test timed out" >> run.log
+  fi
+
+  # A composition that holds resources outside of Docker, such as a Cloud
+  # region, can declare a `ci-cleanup` workflow. It runs here, before the Docker
+  # teardown, with the arguments of the main workflow. A cancelled or timed-out
+  # job reaches this trap through SIGTERM, which ends the mzcompose process
+  # without running the composition's own cleanup path, so this is the only
+  # cleanup such a run gets. A failure goes to run.log, where ci-annotate-errors
+  # turns it into an error annotation and fails an otherwise green job.
+  if echo "$workflows" | grep -x "ci-cleanup" > /dev/null; then
+    ci_unimportant_heading ":docker: Running the composition's ci-cleanup workflow"
+    # The main workflow's containers can outlive its process: SIGTERM ends
+    # mzcompose, but a `docker compose run` container it started keeps going,
+    # and an in-flight `mz region enable` could re-create the region after the
+    # cleanup deleted it. Only this composition's compose project (named after
+    # the composition): other jobs' containers, and containers a composition
+    # drives outside compose, such as kind nodes, keep running for the log
+    # collection below. Killed containers keep their logs for services.log;
+    # `mzcompose down` removes them. Skipped under coverage: the coverage
+    # teardown below stops containers gracefully on purpose, and a kill here
+    # would defeat it.
+    if [ -z "${CI_COVERAGE_ENABLED:-}" ]; then
+      docker ps --quiet --filter "label=com.docker.compose.project=$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" | xargs --no-run-if-empty docker kill > /dev/null || true
+    fi
+    # 15m keeps a hung cleanup from eating the agents' 40-minute cancel grace
+    # period before the artifacts and the error annotation. Normal disables
+    # take about 30 s. The output also goes to run.log so that the artifact
+    # shows why a cleanup failed.
+    if ! bin/ci-builder run "$builder" timeout --signal=TERM --kill-after=30s 15m bin/mzcompose --find "$BUILDKITE_PLUGIN_MZCOMPOSE_COMPOSITION" run ci-cleanup "${run_args[@]:1}" |& sed -u -r 's/\x1B\[[0-9;]*[A-Za-z]//g' | tee -a run.log; then
+      printf "\n%s\n" "$BUILDKITE_LABEL: ci-cleanup workflow failed" >> run.log
+    fi
   fi
 
   ci_unimportant_heading "Post command steps"

ci/spec-sheet/pipeline.template.yml

@@ -111,7 +111,7 @@ steps:
       queue: linux-aarch64-small
 
   - id: cluster-spec-sheet-staging
-    label: "Cluster spec sheet: Staging"
+    label: "Cluster spec sheet: envd Scalability + Cluster Object Count Limits (against Staging)"
     timeout_in_minutes: 3600
     depends_on: devel-docker-tags
     parallelism: 7

ci/test/lint-main/checks/check-mzcompose-files.sh

@@ -59,7 +59,8 @@ check_default_workflow_references_others() {
     )
 
     for file in "${MZCOMPOSE_TEST_FILES[@]}"; do
-      MATCHES_COUNT=$(grep "def workflow_" "$file" -c)
+      # `ci-cleanup` is run by the mzcompose plugin's hook, never by `default`.
+      MATCHES_COUNT=$(grep "def workflow_" "$file" | grep -vc "def workflow_ci_cleanup" || true)
 
       if (( MATCHES_COUNT > 1 )); then
         # mzcompose file contains more than one workflow

misc/python/materialize/cli/ci_annotate_errors.py

@@ -94,6 +94,7 @@
     | SUMMARY:\ .*Sanitizer
     | primary\ source\ \w+\ seemingly\ dropped\ before\ subsource
     | :\ test\ timed\ out
+    | :\ ci-cleanup\ workflow\ failed
     | very\ slow\ coordinator\ message
     # Only notifying on unexpected failures. INT, TRAP, BUS, FPE, SEGV, PIPE
     | \ ANOM_ABEND\ .*\ sig=(2|5|7|8|11|13)

misc/python/materialize/cli/mzcompose.py

@@ -869,7 +869,7 @@ def handle_composition(
                     workflow_name, *args.unknown_subargs[1:], *extra_args
                 )
 
-            if self.shall_generate_junit_report(args.find, composition):
+            if self.shall_generate_junit_report(args.find, workflow_name, composition):
                 junit_suite = self.generate_junit_suite(composition)
                 self.write_junit_report_to_file(junit_suite)
 
@@ -880,8 +880,16 @@ def handle_composition(
                 raise UIError("at least one test case failed")
 
     def shall_generate_junit_report(
-        self, composition_name: str | None, composition: Composition
+        self,
+        composition_name: str | None,
+        workflow_name: str | None,
+        composition: Composition,
     ) -> bool:
+        if workflow_name == "ci-cleanup":
+            # Runs from the CI plugin's exit trap after the main workflow, under
+            # the same BUILDKITE_JOB_ID, so its report would overwrite the main
+            # run's. Its outcome is reported through run.log instead.
+            return False
         if composition.has_testdrive_junit:
             # Testdrive already produced a junit.xml with detailed errors;
             # skip the mzcompose-level junit to avoid duplicate annotations.

test/cluster-spec-sheet/README.md

@@ -4,9 +4,12 @@ Reproduce data for the cluster spec sheet effort.
 
 # Usage
 
-`bin/mzcompose --find cluster-spec-sheet run default`
+`bin/mzcompose --find cluster-spec-sheet run default --target=<target>`
 
 This will run all scenarios currently defined for the cluster spec sheet.
+`--target` is required: `cloud-production`, `cloud-staging`, or `docker`.
+There is deliberately no default, because the CI cleanup below destroys the
+target's region unattended.
 
 Pass `--cleanup` to disable the region after the test.
 
@@ -15,19 +18,24 @@ Pass `--cleanup` to disable the region after the test.
 
 ## Running via Buildkite
 
-The workload runs as part of the release qualification pipeline in Buildkite.
+The workload runs in the `spec-sheet` Buildkite pipeline. After a CI job
+ends, however it ends, the mzcompose plugin runs the `ci-cleanup` workflow
+with the job's arguments; for a Cloud target started with `--cleanup` it
+disables the region. A canceled or timed-out job never reaches the
+composition's own cleanup, so this is what keeps canceled runs from leaving
+regions behind.
 
 ## Running manually in Cloud
 
-To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, you need to set the environment variables `NIGHTLY_CANARY_USERNAME` and `NIGHTLY_CANARY_APP_PASSWORD`.
+To run the cloud canary test manually, you can specify either `--target=cloud-production` (which is hardcoded to aws/us-east-1) or `--target=cloud-staging` (which is hardcoded to aws/eu-west-1). For production, you need to set the environment variables `NIGHTLY_MZ_USERNAME` and `MZ_CLI_APP_PASSWORD`. For staging, the run uses one account from the E2E database pool: set `E2E_STAGING_TEST_FRONTEGG_DATABASE_APP_PASSWORD_<n>` for the pool index `<n>` and select it with `CI_CONCURRENCY_POOL_SLOT=<n>` (outside CI, index 0 is used when the slot is unset); the username is derived from the index. Staging runs also need `BUILDKITE_COMMIT`, which selects the image version to enable.
 
 The username is an email address, the app password is a password generated in the cloud console (something like `mzp_...`).
 
 Once the environment variables have been set, you can run:
 
 ```
 cd test/cluster-spec-sheet
-./mzcompose run default
+./mzcompose run default --target=cloud-production
 ```
 
 ## Running in Docker
@@ -55,11 +63,11 @@ bin/mzcompose --find cluster-spec-sheet run default envd_qps_scalability  --targ
 ```
 or
 ```
-bin/mzcompose --find cluster-spec-sheet run default cluster
+bin/mzcompose --find cluster-spec-sheet run default cluster --target=cloud-production
 ```
 or
 ```
-bin/mzcompose --find cluster-spec-sheet run default envd_objects_scalability
+bin/mzcompose --find cluster-spec-sheet run default envd_objects_scalability --target=cloud-production
 ```
 or
 ```