Skip to content

feat(sentinel): bound cold starts and qualify concurrent game joins - #904

Open
Baylem wants to merge 11 commits into
fix/883-884-mode-scoped-rbacfrom
test/878-wake-join-burst
Open

Baylem wants to merge 11 commits into
fix/883-884-mode-scoped-rbacfrom
test/878-wake-join-burst

Conversation

@Baylem

@Baylem Baylem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Closes #878. Parent tracker #867. This PR targets fix/883-884-mode-scoped-rbac (#900), which includes #886. Both changes touch operator flag registration and Helm arguments. Rollout PR #909 consumes the joined-client interface from this branch.

The operator reserves fleet wake slots durably and updates wake status. Explicit stops take precedence, reconciliation releases old reservations and a restart can recover between reservation and wake acknowledgement. The operations guide covers pool sizing, game-specific join objectives, prewarming and qualification of additional games.

The isolated wake qualification boots Minecraft Java over TCP and Teeworlds over UDP on both Service and Hostport paths. Concurrent game clients join across parked servers while the operator restarts. CI collects join success, retries, p95/p99 and worker CPU, memory and I/O pressure. The uploaded reports apply to those game versions and the tested CI hardware. Current checks

Measured cold-wake qualification

On the single-node Kind runner, all 12 real client joins finished within 300 seconds across two separate Minecraft Java 1.21.4 worlds and Teeworlds 0.7.5. The operator restarted 0.492 seconds into the first burst and 0.600 seconds into the second. A 500 ms fleet sampler observed all three starts in each round and a maximum of one starting StatefulSet at once. The Kubernetes watch recorded one game Pod UID per target per round, zero container restarts and no deletions. All six wake tokens were acknowledged and no lifecycle owner remained.

Round Game and transport Exposure Joined Retries p95 / p99 join
1 Minecraft Java TCP Service 4/4 26 65.822 s / 65.822 s
1 Teeworlds UDP Hostport 2/2 6 79.559 s / 79.559 s
2 Minecraft Java TCP Hostport 2/2 26 56.972 s / 56.972 s
2 Minecraft Java TCP Service 2/2 12 33.386 s / 33.386 s
2 Teeworlds UDP Service 2/2 8 104.878 s / 104.878 s

Peak sampled Kind-node cgroup memory was 4.51 GiB and 4.58 GiB by round. Peak 10-second CPU pressure (PSI some) was 33.54% and 32.32%. Memory PSI some stayed at 0.00% and I/O PSI some peaked at 0.18% and 0.00%. Kubernetes reported no memory, disk or PID pressure. There were two to four clients per cohort, so both nearest-rank percentiles select the slowest observed join. One Teeworlds CR still reported Starting immediately after its first successful join. Executed CI job · Raw reports and admission timeline

Signed-off-by: Baylem <zelathis@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: feature New capability or user-facing addition labels Nov 11, 2026
Signed-off-by: Baylem <zelathis@gmail.com>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

CI Report - b404c20

run 42985473374 - attempt 1

0 failed - 33 passed - 0 skipped - 166m23s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L95.82 S94.18 B88.49 F91.73 L92 S92 B82 F76 +2.18

Changed

Area Files +/-
test/ 10 +871/-5
operator/ 5 +240/-1
docs/ 2 +148/-0
.github/ 1 +17/-0
charts/ 2 +6/-0

Slowest jobs

Job Duration
e2e telemetry / arm64 (kind) 23m29s
build e2e images 15m33s
e2e restricted-workloads / arm64 (kind) 14m59s
e2e capture overhead (kind) 12m46s
build e2e images (arm64) 12m43s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, backup-preparation, bot-fast, multicluster, operator, optional-components, postgres-availability, ratelimit, restricted-workloads, telemetry, upgrade, wake-hostport

…estart

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem changed the base branch from master to fix/native-api-credential-boundary November 11, 2026 06:15
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem changed the base branch from fix/native-api-credential-boundary to fix/883-884-mode-scoped-rbac November 11, 2026 06:18
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
… grants

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem marked this pull request as ready for review November 11, 2026 12:18
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 11, 2026 12:18
@ValgulNecron

Copy link
Copy Markdown
Collaborator

Hey, could you please add changelog entry?

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem

Baylem commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Added the requested unreleased changelog entry.

CI passed on b404c207. The diff through 2c4cb0e6 changes only the changelog entry and its index link. I canceled the repeat CI run because this update changes only Markdown.

@Baylem Baylem self-assigned this Nov 12, 2026
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: feature New capability or user-facing addition

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(sentinel): bound concurrent cold starts and verify client reconnects

2 participants