feat(rollouts): gate fleet template adoption on restored-world canaries

.github/workflows/rollout-canary.yaml

@@ -0,0 +1,21 @@
+name: Rollout canary qualification
+on:
+  pull_request:
+    paths:
+      - "hack/rollout/**"
+      - ".github/workflows/rollout-canary.yaml"
+      - "test/e2e/cmd/join-once/**"
+      - "operator/internal/controller/restore_recovery.go"
+      - "operator/internal/controller/gameserver_template_revision.go"
+permissions:
+  contents: read
+jobs:
+  admission-and-recovery:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - name: Exercise fleet halt, claim identity and rollback decisions
+        run: python3 -m unittest discover -s hack/rollout -p 'test_*.py' -v

changelog/unreleased/909-template-canaries.md

@@ -0,0 +1,5 @@
+# Restored-world template canaries
+
+The rollout tool checks signed template and image pins, restores a stopped backup to a
+separate claim and verifies a game client can join before admitting fleet targets one at a
+time. It rehearses rollback on the clone. [PR #909](https://github.com/GameplanePanel/Gameplane/pull/909)

docs/template-rollouts.md

@@ -62,8 +62,7 @@ kubectl patch gameserver my-server -n gameplane-games --type=merge \
 ```
 
 An old request never approves a newer digest. Requests can be made one server at
-a time or by an administrator's batch process. There is no built-in canary,
-maintenance-window or fleet concurrency scheduler. Once admitted, an update
+a time or through the restored-world fleet procedure below. Once admitted, an update
 finishes using that exact snapshot even if another catalog update arrives or
 the policy changes during shutdown.
 
@@ -131,3 +130,84 @@ content as reconstruction prerequisites. CSI
 restores copy the source server's effective template at restore planning time,
 not necessarily its template at backup time. Select a compatible Module version
 and `OnRequest` policy before recovery if automatic adoption would be unsafe.
+
+## Restored-world fleet procedure
+
+Use a dedicated Module installation for one fleet. Put **every** server
+referencing this cluster-scoped template under `OnRequest` before publishing
+a new bundle. Keep both bundle and game image in the registry. The admin
+runner needs Python 3, kubectl, cosign, a trusted *game image* public key and
+a join client for the game. Build the Minecraft Java/Terraria client from
+the platform checkout with `go build -o join-once ./test/e2e/cmd/join-once`.
+For an authenticated or proprietary game, supply a trusted entitled client
+that implements the same CLI and returns `VERDICT<TAB>PASS<TAB>JOINED<TAB>evidence`.
+The bundled Minecraft client expects offline-mode test servers.
+The plan's `port` and `protocol` must name a game-container port. TCP (the
+default) uses a port-forward to that specific Pod. For a UDP game set
+`"protocol": "UDP"` and run its qualified join client on a host or in a Pod
+that can route to the game's Pod IP. Permit this runner's traffic to the
+declared UDP port in the selected workload's ingress policy.
+
+Create a stopped, all-data Restic backup of a representative server. Create
+and start a **separate** GameServer with the same template and version, restore
+that backup into it, and wait for the Restore to succeed. Give each included
+data or extra volume a persistent game marker. For excluded rebuildable mod
+files, verify the reconstructed content or its expected absence on the target.
+The runner compares the PVC UIDs in the Restore plan with the live clone,
+records all owned claims around every adoption and compares marker and game
+save command output exactly. Use a trusted plan with argument arrays that
+check the actual data in each listed claim. A server's checks must account for
+every retained claim. Use the server name in each real claim name.
+
+An example plan for three Minecraft servers, each with only its primary PVC:
+
+```json
+{
+  "namespace": "games", "module": "survival-fleet", "source": "survival-a",
+  "backup": "pre-upgrade", "restore": "clone-from-pre-upgrade",
+  "rollback": "restore-required", "game": "minecraft-java", "port": 25565,
+  "baseline": {"revision": "<64-hex-template-revision>", "bundle": "sha256:<64-hex>", "image": "registry.example/game@sha256:<64-hex>"},
+  "candidate": {"revision": "<64-hex-template-revision>", "bundle": "sha256:<64-hex>", "image": "registry.example/game@sha256:<64-hex>"},
+  "canary": {"name": "survival-clone", "checks": [{"claim": "survival-clone-data", "argv": ["cat", "/data/fleet-marker"], "expected": "world-a"}]},
+  "targets": [
+    {"name": "survival-a", "checks": [{"claim": "survival-a-data", "argv": ["cat", "/data/fleet-marker"], "expected": "world-a"}]},
+    {"name": "survival-b", "checks": [{"claim": "survival-b-data", "argv": ["cat", "/data/fleet-marker"], "expected": "world-b"}]}
+  ],
+  "save": {"argv": ["rcon-cli", "save-all flush"], "expected": "Saving the game (this may take a moment!)Saved the game"}
+}
+```
+
+Use the exact positive acknowledgment for your qualified game version and
+locale. The runner follows the server order in the target list.
+
+```sh
+opts=(fleet.json --context games-production --image-key keys/game.pub --join ./join-once --timeout 900)
+python3 hack/rollout/canary.py preflight "${opts[@]}"
+# Install the signed, digest-pinned candidate Module through GitOps; await Ready.
+python3 hack/rollout/canary.py canary "${opts[@]}"
+# Reinstall the signed, digest-pinned baseline Module; await Ready.
+python3 hack/rollout/canary.py rollback "${opts[@]}"
+# Restore the exact signed candidate Module; await Ready.
+python3 hack/rollout/canary.py promote "${opts[@]}"
+```
+
+Preflight verifies the installed baseline and full inventory across namespaces.
+The canary must boot, join and save on its restored old data, adopt the exact
+candidate revision, then join, validate markers and save again. A compatible
+rollback must boot the baseline on the migrated save. For `restore-required`,
+the runner requests a suspended baseline and waits for its pod to disappear
+before restoring the original snapshot and starting it. The
+production targets stay on the baseline throughout this rehearsal. Promotion
+requalifies the candidate on the clone and admits one fleet member at a time.
+Each member must retain its PVCs and pass the same join, marker and save checks.
+
+Progress, active target, PVC UIDs, image, revision and individual receipts
+live in a namespace ConfigMap named `gp-rollout-<module-name-hash>`.
+Resource-version updates serialize runs independently of the GameServer owner.
+After a failure, inspect the journal and target before retrying the same phase
+with the same plan. After a runner crash, verify that process has exited before
+supplying its exact recorded token with `--recover-owner`. Archive the completed
+journal before deleting it to start a different plan. Restrict this runner to
+trusted release operators: it reads cluster-wide GameServers and module state,
+reads namespaced pods/workloads/PVCs, patches named GameServers, creates clone
+Restores and journal ConfigMaps, and uses namespaced pod exec and port-forward.