Skip to content

fix(operator): recover failed game pods after a corrected rollout - #944

Open
Baylem wants to merge 3 commits into
masterfrom
fix/recover-corrected-game-rollout
Open

Baylem wants to merge 3 commits into
masterfrom
fix/recover-corrected-game-rollout

Conversation

@Baylem

@Baylem Baylem commented Nov 12, 2026

Copy link
Copy Markdown
Member

Recover an owned, failed game pod when its rolling StatefulSet has observed a different target revision. The controller checks active lifecycle operations and stop fences, then requests a namespace-scoped eviction with UID and resource-version preconditions. Disruption budgets can defer recovery.

The controller suite exercises image-pull and crash-loop failures, unrelated objects, recovery fences and disruption-budget retries. Adds an operator e2e scenario that deploys an invalid image, restores the original image, and waits for a healthy replacement ordinal.

Closes #934.

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 12, 2026 05:18
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 24 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between e3826d8 and fc8ded3.

Files selected for processing: 10

  • changelog/unreleased/934-statefulset-rollout-recovery.md
  • charts/gameplane/templates/operator.yaml
  • operator/config/rbac/role_namespace.yaml
  • operator/internal/controller/gameserver_controller.go
  • operator/internal/controller/gameserver_rollout_recovery.go
  • operator/internal/controller/gameserver_rollout_recovery_test.go
  • operator/internal/controller/lifecycle_operation.go
  • operator/specs.md
  • test/e2e/buckets.sh
  • test/e2e/template_rollout_e2e_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

CI Report - 66ea4bb

run 43161320932 - attempt 1

0 failed - 13 passed - 0 skipped - 20 cancelled - 18m44s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L94.03 S92.14 B86.51 F90.19 L92 S92 B82 F76 +0.14

Changed

Area Files +/-
operator/ 6 +401/-4
test/ 2 +52/-0
charts/ 1 +4/-0
changelog/ 1 +3/-0

Slowest jobs

Job Duration
go (gp-module / amd64) 15m6s
go (svcutil / amd64) 15m6s
go (tunnel / amd64) 15m6s
go (tunnel / arm64) 15m6s
lint (api) 15m5s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, bot-fast, multicluster, operator, optional-components, restricted-workloads, telemetry, upgrade

@Baylem Baylem left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified recovery against a disposable Kubernetes cluster. A healthy server received an invalid image, entered image pull backoff, and returned to Ready on a new pod UID after restoring the original image. The scenario passed in 25.21 seconds.

Reviewed the shared lifecycle claim, pod UID and resource-version preconditions, and cleanup after a lost eviction response. Tests exercise a competing backup claim, disruption-budget refusal, a changed server generation, and controller restart. The operator retains ownership through termination of the old pod.

@Baylem Baylem self-assigned this Nov 12, 2026
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recover failed game pods after a corrected StatefulSet rollout

1 participant