Skip to content

fix(operator): schedule service wakers and tunnels on approved workers - #910

Merged
ValgulNecron merged 15 commits into
feat/875-game-worker-policyfrom
fix/876-auxiliary-worker-placement
Nov 12, 2026
Merged

ValgulNecron merged 15 commits into
feat/875-game-worker-policyfrom
fix/876-auxiliary-worker-placement

Conversation

@Baylem

@Baylem Baylem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Closes #876. Parent tracker #867. This PR targeted feat/875-game-worker-policy and depended on #903. After #917 merged, ValgulNecron merged both placement PRs in stack 931.

Service-backed wakers and the FRP, Tailscale and playit relays use an administrator-approved auxiliary worker pool. Hostport wakers keep the game's placement and recorded endpoint node. Removing the override reconciles the generated workloads to their inherited placement.

The management-placement CI drill uses tainted game and management workers and a tainted control plane. It drains the relay, checks the scheduler's Unschedulable condition, restores the relay, and performs real wake handoffs through ClusterIP, NodePort, MetalLB ingress and Hostport using populated local-path and CSI claims. Provider health endpoints use an isolated test image. The reference CSI driver and its provisioner are pinned to upstream commits and confined to the CI cluster. Current PR checks

Live topology qualification

The 193.32-second drill at e6df3b81 scheduled all three relay providers on the management worker, evicted the relay and observed its unschedulable replacement, then preserved each populated local-path and CSI world through all four wake exposures. The local-path PVC retained UID 621e5ad3-be56-4095-a6f8-bc64fd78b8cd and the CSI PVC retained UID 2e630082-0996-4b91-85c2-e9ad88255ccf. A separate inherited tenant memory fixture failed in the same job. Executed job

The extended drill passed in 202.15 seconds on d3c5672. Draining the single data worker evicted both games. Both replacement Pods remained Unschedulable while the management relay stayed ready on the other worker. After uncordon, both games recovered their original world markers and PVC/PV UIDs, five seconds after the drain began. The local-path PVC retained UID 3d071d39-9ba9-4461-854d-9d20b2d60426 and the CSI PVC retained UID 64bce869-94f9-4ed8-92d9-b1e5357ff0db. All eight wake handoffs also passed across the two storage types and four exposures. A separate, inherited workload-policy fixture failed before making its Kubernetes request. The signed child integration at cea0609 includes the parent correction. Executed data-worker drain · Current PR checks

Shared CI and worker-policy integration

The signed integration at d958210a contains #903’s kernel-backed tenant memory check and #917’s shared browser bootstrap. Its management-placement lane exercises both the inherited worker policy and the local-path/CSI data-worker drain with eight wake handoffs. Initial CI run

Combined data-worker drain and tenant isolation on d958210

On d958210a, the 202.92-second Kind drill completed all eight wake handoffs and drained the only data worker. Both games’ replacements remained unschedulable while the management relay stayed ready. After the worker was uncordoned, each game recovered its original world marker five seconds from the start of the drain. The local-path claim and volume retained UIDs 9b88d02e-83ce-4502-8dda-6cbccf7b0a5e and 0b741f5e-37d0-4164-aca6-5decd9f8f7e6. The CSI claim and volume retained UIDs 5e1e1e5d-7e36-4e05-bf78-26cf93f25d5a and bc5d3e4a-1e0c-4e4d-938e-ae53e062ad0e.

The inherited worker-policy drill also passed in 107.85 seconds. The tenant container was OOMKilled at 32 MiB, with Pod max/oom/oom_kill counters increasing from 0/0/0 to 22/1/2. The neighboring Minecraft Pod retained its identity with zero restarts. Its baseline, concurrent and post-OOM p95/p99 tick durations were 0.3/1.5 ms, 0.2/3.6 ms and 0.1/0.3 ms. Restic completed backup and repository read verification in 20.90 seconds. Combined worker and drain job

The reviewed integration 694264df passed all eight wake handoffs and the populated data-worker drain in 213.83 seconds. The local-path game recovered its original world marker and PVC/PV four seconds after drain began. The CSI game recovered its own marker and PVC/PV in five seconds. The local-path PVC/PV UIDs were 8c31cd04-9353-45e2-9e32-fbe46dde8291 / cda65141-99e2-4473-80ee-bd404a218da5. The CSI PVC/PV UIDs were 91503a8c-dc69-4a77-b80c-c43ba8df249f / 54699f0c-bd88-4519-aa6c-c79f43d0e347.

The concurrent worker-policy drill passed in 102.01 seconds. Kubernetes reported Error/137 at the 32 MiB container limit. The identified Pod's max/oom/oom_kill counters rose from 0/0/0 to 21/1/2. Neighbor Minecraft p95/p99 tick time was 0.2/0.6 ms before pressure, 0.2/2.1 ms during pressure and 0.2/0.2 ms after the OOM. Restic backed up the game and read-verified the repository in 19.06 seconds. Exact-head placement drill · Full hosted workflow.

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 27 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 4cbb65a and 694264d.

Files selected for processing: 15

  • changelog/unreleased/910-auxiliary-placement.md
  • charts/gameplane/templates/operator.yaml
  • charts/gameplane/values.yaml
  • docs/production-placement.md
  • operator/cmd/main.go
  • operator/internal/controller/auxiliary_placement_test.go
  • operator/internal/controller/game_workload_policy.go
  • operator/internal/controller/gameserver_controller.go
  • operator/internal/controller/gameserver_sentinel.go
  • operator/internal/controller/gameserver_sentinel_envtest_test.go
  • operator/internal/controller/gameserver_tunnel.go
  • test/e2e/auxiliary-placement.sh
  • test/e2e/auxiliary_placement_e2e_test.go
  • test/e2e/buckets.sh
  • test/e2e/suite-components.json
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: fix Bug fix labels Nov 11, 2026
Signed-off-by: Baylem <zelathis@gmail.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

CI Report - 694264d

run 43112890031 - attempt 1

0 failed - 33 passed - 0 skipped - 119m13s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L94.03 S92.14 B86.51 F90.19 L92 S92 B82 F76 +0.14

Changed

Area Files +/-
test/ 4 +265/-0
operator/ 7 +129/-11
docs/ 1 +29/-2
charts/ 2 +6/-0
changelog/ 1 +4/-0

Slowest jobs

Job Duration
e2e telemetry / arm64 (kind) 23m44s
build e2e images 16m58s
e2e operator / amd64 (kind) 15m4s
e2e restricted-workloads / arm64 (kind) 14m44s
e2e capture overhead (kind) 12m58s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, bot-fast, multicluster, operator, optional-components, postgres-availability, ratelimit, restricted-workloads, telemetry, upgrade, wake-hostport

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem added this pull request to stack #914 November 11, 2026 10:06
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem added this pull request to stack #918 November 11, 2026 16:22
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem marked this pull request as ready for review November 11, 2026 17:48
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 11, 2026 17:48
@ValgulNecron

Copy link
Copy Markdown
Collaborator

Hey, could you please add changelog entry?

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem

Baylem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member Author

Added the requested unreleased changelog entry.

CI passed on fc5b25dd. The diff through a01b9503 changes only the changelog entry and its index link. I canceled the repeat CI run because this update changes only Markdown.

The subsequent integration at 694264df brings in the legacy-game drain fix from #903 and the probe validator from #889. The Linux regressions passed for auxiliary placement, legacy and already-pinned games, and rejection of unsafe probes in stored revisions. Full CI on the combined code.

Signed-off-by: Baylem <zelathis@gmail.com>
ValgulNecron
ValgulNecron previously approved these changes Nov 12, 2026
@Baylem
Baylem removed this pull request from stack #918 November 12, 2026 02:11
@Baylem
Baylem added this pull request to stack #931 November 12, 2026 02:11
@Baylem
Baylem dismissed ValgulNecron’s stale review November 12, 2026 02:11

The merge-base changed after approval.

@ValgulNecron
ValgulNecron merged commit e3826d8 into master Nov 12, 2026
113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(operator): schedule service-backed sentinels and tunnels on tainted workers

2 participants