fix(operator): schedule service-backed sentinels and tunnels on tainted workers #876

Production readiness audit of Gameplane v0.4.0, commit 07bee47b. Tracking issue and PR merge order: #867.

Finding

Hostport sentinels inherit game placement. Service-backed sentinels clear the game's scheduling constraints and tunnel deployments lack equivalent placement options. With every eligible worker tainted, a game can lose its wake listener or relay.

Work

Provide administrator-controlled auxiliary placement for service-backed sentinels and each supported tunnel. Retain hostport co-location/handoff constraints. Specify behavior when no node satisfies the policy and give operators a useful condition or event.

Acceptance criteria

Use tainted game and management pools and a tainted control plane. Verify each exposure mode, wake handoff, relay restart, a worker drain and local/CSI PVC topology. Confirm workloads land only on intended nodes and an unschedulable configuration is diagnosable.

PR ownership and prior work

Consume H08’s approved placement logic and fixtures where the same policy applies. This PR owns sentinel/tunnel builders and their exceptions. Hostport node identity and handoff were addressed in #730 and #840.

Sentinel builder · Tunnel builder · Documented taints

Executed relay and storage qualification

The 193.32-second qualification placed all three relay providers on the management worker. Draining that worker made the relay replacement Unschedulable until recovery. A populated local-path game and a populated CSI game then slept and woke through ClusterIP, NodePort, MetalLB VIP and Hostport, retaining their original volume claims and world markers. The relay containers used a health fixture. The separate worker-quota test in that job failed on its memory-leak fixture. Executed placement job

The extended drill passed in 202.15 seconds on d3c5672. Draining the single data worker evicted both games. Both replacement Pods remained Unschedulable while the management relay stayed ready on the other worker. After uncordon, both games recovered their original world markers and PVC/PV UIDs, five seconds after the drain began. The local-path PVC retained UID 3d071d39-9ba9-4461-854d-9d20b2d60426 and the CSI PVC retained UID 64bce869-94f9-4ed8-92d9-b1e5357ff0db. All eight wake handoffs also passed across the two storage types and four exposures. A separate, inherited workload-policy fixture failed before making its Kubernetes request. The signed child integration at cea0609 includes the parent correction. Executed data-worker drain · Current PR checks

Combined data-worker drain and tenant isolation on d958210

On d958210a, the 202.92-second Kind drill completed all eight wake handoffs and drained the only data worker. Both games’ replacements remained unschedulable while the management relay stayed ready. After the worker was uncordoned, each game recovered its original world marker five seconds from the start of the drain. The local-path claim and volume retained UIDs 9b88d02e-83ce-4502-8dda-6cbccf7b0a5e and 0b741f5e-37d0-4164-aca6-5decd9f8f7e6. The CSI claim and volume retained UIDs 5e1e1e5d-7e36-4e05-bf78-26cf93f25d5a and bc5d3e4a-1e0c-4e4d-938e-ae53e062ad0e.

The inherited worker-policy drill also passed in 107.85 seconds. The tenant container was OOMKilled at 32 MiB, with Pod max/oom/oom_kill counters increasing from 0/0/0 to 22/1/2. The neighboring Minecraft Pod retained its identity with zero restarts. Its baseline, concurrent and post-OOM p95/p99 tick durations were 0.3/1.5 ms, 0.2/3.6 ms and 0.1/0.3 ms. Restic completed backup and repository read verification in 20.90 seconds. Combined worker and drain job

The reviewed integration 694264df passed all eight wake handoffs and the populated data-worker drain in 213.83 seconds. The local-path game recovered its original world marker and PVC/PV four seconds after drain began. The CSI game recovered its own marker and PVC/PV in five seconds. The local-path PVC/PV UIDs were 8c31cd04-9353-45e2-9e32-fbe46dde8291 / cda65141-99e2-4473-80ee-bd404a218da5. The CSI PVC/PV UIDs were 91503a8c-dc69-4a77-b80c-c43ba8df249f / 54699f0c-bd88-4519-aa6c-c79f43d0e347.

The concurrent worker-policy drill passed in 102.01 seconds. Kubernetes reported Error/137 at the 32 MiB container limit. The identified Pod's max/oom/oom_kill counters rose from 0/0/0 to 21/1/2. Neighbor Minecraft p95/p99 tick time was 0.2/0.6 ms before pressure, 0.2/2.1 ms during pressure and 0.2/0.2 ms after the OOM. Restic backed up the game and read-verified the repository in 19.06 seconds. Exact-head placement drill · Full hosted workflow.