Skip to content

feat(rollouts): gate fleet template adoption on restored-world canaries - #909

Open
Baylem wants to merge 18 commits into
stack/881-backup-and-wakefrom
feat/881-staged-template-rollouts
Open

Baylem wants to merge 18 commits into
stack/881-backup-and-wakefrom
feat/881-staged-template-rollouts

Conversation

@Baylem

@Baylem Baylem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Closes #881. Parent tracker #867.

hack/rollout/canary.py verifies a signed, pinned Module and independently verifies the pinned game image. Preflight checks every GameServer using the dedicated installation. It qualifies a stopped all-data Restic clone, checks a pod-bound client join, save markers and claim UIDs, rehearses compatible or restore-required rollback on the clone, then requalifies it and admits one fleet target at a time. A resourceVersion-protected ConfigMap journals the active target and receipts outside the per-server lifecycle owner. See docs/template-rollouts.md for the four GitOps phases and recovery.

Validation includes decision tests for a failed join, wrong digest, changed UIDs, stale journals and per-target halts. TestGameServer_RestoredRolloutRehearsal uses the real operator, Restic, kubelet and PVCs in Kind. The Kind rehearsal tests an unpullable image and a running HTTP-only candidate that rejects a real Minecraft handshake. It checks that fleet admission halts, exercises compatible and restore-required rollback, verifies primary and extra save markers, and retains a different rebuildable mod file on each server. The runner supports TCP Pod port-forwarding and UDP client joins to the verified Pod IP from an in-cluster or routed runner. The Minecraft bot in #907 exercises a real game join and save on restored data. An entitled game and production signer matrix remains deployment-specific.

The upstream integration base stack/881-backup-and-wake contains #907 (qualified backups/restores) and #904 (join client). Land the prerequisite PRs, retarget this PR to master and rerun CI. No shared cluster was deployed.

Restored rollout qualification on ae14d5ab74b4b386688914f267d46bd1571db534

The Kind rehearsal passed all four cases on ARM64 (543.65s) and AMD64 (479.37s). Independent AMD64 rehearsal The compatible rollback kept the save readable directly. The format-changing rollback used Restic snapshot b63139d0bcf881bf22167bab7d98d562ef8bdb7b0931cfc3fe933c3350a4d282 in Restore 6acaa54e-c9f2-4333-b8ca-43b505e7b5f6, removed the candidate-only format marker, and returned the primary and extra world values. The excluded mod file retained the clone's own clone-package value.

Clone PVC Before candidate After candidate and rollback
Primary world fe45d715-bcca-4cee-9d30-3eda32546320 fe45d715-bcca-4cee-9d30-3eda32546320
Extra world 2e9c6e49-d664-4817-9367-7754dab0aa27 2e9c6e49-d664-4817-9367-7754dab0aa27
Rebuildable mods 32d84e79-8484-446b-b31c-61bf29ba7efb 32d84e79-8484-446b-b31c-61bf29ba7efb

A running HTTP-only candidate served the world marker but failed the real Minecraft handshake. The journal kept only its baseline receipt and blocked promotion. An unpullable candidate also halted the rollout. The untouched fleet member retained its original pod UID throughout. Kind rehearsal and raw receipts · Fleet admission decision tests · Successful real Minecraft join and save on a restored world in the backup parent

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: feature New capability or user-facing addition labels Nov 11, 2026 — with ChatGPT Codex Connector
…n parents

Signed-off-by: Baylem <zelathis@gmail.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

CI Report - ae14d5a

run 42985674820 - attempt 1

0 failed - 33 passed - 0 skipped - 180m21s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L95.82 S94.18 B88.49 F91.73 L92 S92 B82 F76 +2.18

Changed

Area Files +/-
hack/ 2 +657/-0
test/ 4 +248/-0
docs/ 1 +82/-2
.github/ 1 +21/-0
operator/ 1 +5/-0

Slowest jobs

Job Duration
e2e telemetry / arm64 (kind) 22m42s
e2e restricted-workloads / arm64 (kind) 14m48s
build e2e images 14m31s
e2e web live / amd64 (kind) 14m25s
e2e capture overhead (kind) 12m37s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, backup-preparation, bot-fast, management-placement, monitoring, multicluster, network-isolation, operator, optional-components, postgres-availability, ratelimit, recovery, restricted-workloads, telemetry, upgrade

…hearsals

Signed-off-by: Baylem <zelathis@gmail.com>
…cation

Signed-off-by: Baylem <zelathis@gmail.com>
…lification

Signed-off-by: Baylem <zelathis@gmail.com>
…stored claims

Signed-off-by: Baylem <zelathis@gmail.com>
…orkload

Signed-off-by: Baylem <zelathis@gmail.com>
…resources

Signed-off-by: Baylem <zelathis@gmail.com>
…on parent

Signed-off-by: Baylem <zelathis@gmail.com>
…llouts

Signed-off-by: Baylem <zelathis@gmail.com>
…ixes

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem marked this pull request as ready for review November 11, 2026 11:22
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 11, 2026 11:22
@ValgulNecron

Copy link
Copy Markdown
Collaborator

Hey, could you please add changelog entry?

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem

Baylem commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Added the requested unreleased changelog entry.

CI passed on ae14d5ab. The diff through 6c537267 changes only the changelog entry and its index link. I canceled the repeat CI run because this update changes only Markdown.

@Baylem Baylem self-assigned this Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: feature New capability or user-facing addition

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(operator): gate template rollouts on restored-world canaries and successful joins

2 participants