feat(operator): enforce tenant resource bounds and game-worker placement #875

Production readiness audit of Gameplane v0.4.0, commit 07bee47b. Tracking issue and PR merge order: #867.

Finding

A partial server resource override replaces the template resource object, which can remove requests or a memory ceiling. User-provided selectors and broad tolerations can bypass a pool designed only around taints.

Work

Resolve effective resources without silently losing required bounds. Add an administrator-controlled worker placement and resource policy, enforce it for direct CR and panel-created games, and provide per-tenant CPU, memory, pod and PVC quota guidance/configuration. Qualify reserve capacity for starts, backups and worker loss. Preserve measured CPU bursting for supported Burstable tiers.

Acceptance criteria

Test partial overrides, missing limits, an unrestricted Exists toleration, conflicting selectors, existing CRs and a drained worker. Run a leaking workload to the tenant ceiling and verify worker placement, neighboring games’ tick latency, scheduler behavior and headroom.

PR ownership and prior work

Own effective placement validation and fixtures. H09 consumes them for auxiliary workloads. Share tenant namespace provisioning with H02 where applicable. Reuse the existing management placement and auxiliary requests from #762. Burstable game QoS is intentional (#815).

Game pod scheduling · Resource resolution · Placement profile

Qualification evidence

PR #903 passed CI at commit 3a59a6c3. After the original worker was lost, the leaking tenant hit its 32 MiB ceiling on the surviving worker. The neighboring Minecraft Java process kept the same Pod UID with zero restarts. Its p99 tick was 0.4 ms at baseline, 0.4 ms during the concurrent replacement start, leak and backup, and 3.1 ms after the tenant OOM. Restic completed its backup and repository read verification in 16.65 seconds. The surviving four-CPU runner had 695m CPU and 1,426 MiB of memory requested. Executed placement drill · PR and complete results

Current-head worker policy qualification

The full worker test passed in 104.52 seconds on a2def5e. Its live operator-policy subtest took 35.89 seconds. A pre-existing 64 MiB server stopped after the administrator enabled a 32 MiB ceiling. The original StatefulSet kept its Pod template. A newly submitted invalid GameServer produced no StatefulSet, while a valid 50m CPU override retained its 16 MiB memory request and 32 MiB limit on the selected worker. Both amd64 and arm64 operator tests also cover graceful stop with the pinned template and preservation of foreign StatefulSets.

After loss of the original worker, a single-process tenant leak was OOMKilled at 32 MiB. The neighboring Minecraft Pod kept its UID with zero restarts. Baseline, concurrent workload and post-OOM p95/p99 tick times were 0.2/2.3 ms, 0.2/0.5 ms and 0.2/0.3 ms. Restic completed backup and repository read verification in 18.26 seconds. The surviving worker had 4 CPUs and 16,366,640 KiB allocatable memory. Scheduled requests totaled 585m CPU and 1,346 MiB memory. Executed placement drill

Measured tenant cgroup isolation on 7a6b951

The live Kind worker-policy drill passed in 110.04 seconds on 7a6b951e, including the 38.54-second operator reconciliation case. The 32 MiB game container in Pod 2a2d3c90-2949-4d65-afd9-b2c5d852132d exited 137 with Kubernetes reason OOMKilled. In that Pod’s own cgroup, max, oom and oom_kill rose from 0/0/0 to 37/1/2. The replacement container ran in the same Pod. The neighboring Minecraft server retained its Pod UID with zero restarts. Its p95/p99 tick durations were 0.2/0.6 ms before load, 0.2/2.3 ms during worker loss, leakage and backup, then 0.1/0.2 ms after the kill. Restic backed up and read-verified its repository in 21.32 seconds. Scheduled requests on the surviving 4-CPU worker were 595m CPU and 1,362 MiB memory. Executed placement drill

The drill passed again on the final integration 8aa6ed5b in 106.96 seconds, including 38.28 seconds of live operator reconciliation. The 32 MiB tenant was OOMKilled and its Pod counters rose from 0/0/0 to 39/1/2. The neighboring Minecraft Pod kept its UID with zero restarts. Its baseline, concurrent and post-OOM p95/p99 tick durations were 0.2/0.6 ms, 0.2/2.4 ms and 0.2/0.4 ms. Restic backed up and read-verified the repository in 19.11 seconds. Requests on the surviving worker totaled 585m CPU and 1,346 MiB memory. Final placement drill · Final integration checks

The dependent branch also exercised Kubernetes reason Error with exit 137 on the same fixture. In that Pod’s memory cgroup, max/oom/oom_kill rose from 0/0/0 to 40/1/2 at the 32 MiB limit. Dependent run with generic Kubernetes termination reason

Worker-policy qualification after review

The signed integration 4cbb65a8 passed the native Kind placement drill in 101.74 seconds, including 37.01 seconds of GameServer reconciliation. The test required the worker admission policy to deny a 64 MiB game container even though its tenant quota allowed 64 MiB. It also verified that a second real Pod hit the tenant quota. Both native operator unit suites passed, including regressions for graceful first-pin migration, a stale scale-up racing a surviving world writer, and rejection of an unsafe historical probe before draining. Placement and admission drill · Operator amd64 · Operator arm64

The 32 MiB tenant exited 137 with reason OOMKilled. Its Pod cgroup counters for max, oom and oom_kill rose from 0/0/0 to 26/1/2. Neighboring Minecraft retained its Pod UID with zero container restarts. Its p95/p99 tick times were 0.4/3.4 ms before load, 0.2/0.3 ms during worker loss, leak and backup, and 0.2/2.8 ms after the tenant OOM. The worker carried 595m CPU and 1,362 MiB in requests. Restic completed the backup and read-verified the repository in 17.23 seconds. Measured drill

The full CI report passed on 4cbb65a8, including the admission-specific Kind test and the operator suites for both architectures. Final CI report