Repository navigation
chore(audit): track production readiness findings and PR merge order #867
Description
Coordinate the 10 High, 6 Medium and 2 Low findings from the November 10 production readiness audit of Gameplane v0.4.0 and its official game modules.
Finding checklist
High
- H01 · fix(operator): restrict tenant-defined probes to the game pod #868 — Restrict tenant-supplied Kubernetes probes to the game pod
- H02 · fix(networking): isolate tenant traffic across games, sentinels and tunnels #869 — Constrain tenant network access across games, sentinels and tunnels
- H03 · fix(backups): distinguish flushed saves from verified write freezes #870 — Distinguish a flushed live save from a verified frozen backup
- H04 · fix(backups): verify that the original game resumes saving before clearing cleanup #871 — Keep backup cleanup pending until the original game confirms saving resumed
- H05 · fix(chart): read PostgreSQL credentials from a Secret for API startup and bootstrap #872 — Read PostgreSQL DSNs from a referenced Secret in API deployments and bootstrap
- H06 · fix(api): bound audit writes and lock waits during PostgreSQL stalls #873 — Bound audit persistence during a PostgreSQL stall
- H07 · feat(api): qualify active-active PostgreSQL panels and fenced recovery #874 — Qualify PostgreSQL recovery and remove single-writer limits only after HA invariants hold
- H08 · feat(operator): enforce tenant resource bounds and game-worker placement #875 — Enforce tenant resource bounds and approved game-worker placement
- H09 · fix(operator): schedule service-backed sentinels and tunnels on tainted workers #876 — Make service-backed sentinels and tunnel pods schedulable on tainted worker pools
- H10 · test(release): verify published artifacts on fresh installs and upgrades #877 — Qualify publication, install and upgrade of the release following the failed v0.4.0 run
Medium
- M01 · feat(sentinel): bound concurrent cold starts and verify client reconnects #878 — Qualify wake-on-connect under simultaneous cold starts and client retries
- M02 · feat(tunnel): measure relay saturation, packet loss and latency #879 — Expose relay saturation and qualify UDP loss, jitter and latency against direct traffic
- M03 · feat(observability): discover tenant metrics and alert on missing game signals #880 — Discover every tenant’s metrics and distinguish missing game performance signals
- M04 · feat(operator): gate template rollouts on restored-world canaries and successful joins #881 — Gate OCI template rollouts on restored-world canaries and successful game joins
- M05 · fix(chart): stabilize offline GitOps renders and preserve trust and PVCs on upgrade #882 — Make offline GitOps renders deterministic and qualify CRD, trust and PVC upgrades
- M06 · fix(rbac): isolate signing authority and narrow administrative permissions #883 — Separate agent signing authority and scope optional administrative Kubernetes permissions
Low
- L01 · refactor(rbac): align API ClusterRole names with their permissions #884 — Align API ClusterRole names and documentation with the verbs they grant
- L02 · docs(monitoring): clarify per-server and legacy probe ingress rules #885 — Correct monitoring guidance for per-server and legacy probe ingress policies
PR execution plan
Open one review for a whole finding, or for the explicitly paired findings below. Each review includes its implementation and acceptance tests or required field evidence. Each of the 18 findings has a linked review, including the two paired reviews. The separately maintained catalog pointer in #926 delivers the two merged game-module changes.
Review each child against its stated base. Merge the parent PRs into master first, retarget the child to master, check Files changed for the child scope, and rerun CI. If a parent was squash-merged, merge current master into the child and verify that the parent’s changes disappear from its three-dot diff. Use a signed follow-up merge rather than rewriting published commits.
Upstream branches and merge order
| Issue | Upstream Gameplane branch | Review, dependencies and qualification |
|---|---|---|
| #868 | fix/pod-local-probes | #889 → master. Owns probe-host admission and stored-CR handling. |
| #869 + #885 | fix/tenant-network-profile | #894 → combined base. Merge #886 and #889 first. One Cilium suite owns the tenant matrix and the probe-guide examples. |
| #870 | fix/870-backup-consistency | #907 → #900 via the combined base. #899 and #889 merged. #900 supplies the signed one-request webhook fixture fix required by the operator qualification. The same base consumes the tenant-scoped placement fixture from #886. At 30db243f, the review still contains exactly 50 backup-consistency files. The engine declarations merged in module #59. #926 delivers the catalog pin used by the panel and publisher. Built from fix/871-durable-backup-resume. Restore and join a changing world, verify consistency classifications, and coordinate the engine modules. |
| #871 | fix/871-durable-backup-resume | #899 → master. Owns the agent/operator acknowledgment contract. Supersedes fork PR #893. |
| #872 | fix/postgres-dsn-secret | #887 → #886. Both edit the API Deployment’s environment configuration. |
| #873 | fix/audit-db-backpressure | #888 → master. Owns bounded audit writes and the PostgreSQL blackhole fixture. |
| #874 | feat/874-postgres-recovery-ha | #911 → combined base of #887, #888, #904, #917 and #922 (includes #900 and #886). The API Deployment and operator flags combine security, RBAC, wake and HA settings; the PostgreSQL qualification has its own suite selection. The product decision is active-active native panel APIs backed by one writable PostgreSQL primary. Qualify fenced WAL recovery, concurrent writers, mixed rollout, and replica, node and database loss. SQLite remains single-writer. |
| #875 | feat/875-game-worker-policy | #903 merged into master with #910 in native stack 931 (e3826d8d). The browser prerequisite #917 and probe validator #889 have merged. The worker migration checks the stored template before replacing a legacy game Pod and waits for the previous volume writer to exit. Qualification covers tenant ceilings, neighboring game latency and capacity during starts, backups and a worker drain. |
| #876 | fix/876-auxiliary-worker-placement | #910 and #903 merged into master together in native stack 931 (e3826d8d). Built from feat/875-game-worker-policy. Exercise relay and sentinel placement, wake handoff and PVC topology on tainted pools. |
| #877 | test/877-published-release-consumer | #906 → combined base of #901 and #917. Reuses the populated-upgrade fixture and shared browser CI. The anonymous consumer passed a fresh v0.4.0 install and populated v0.3.0 upgrade against published artifacts. Release evidence. |
| #878 | test/878-wake-join-burst | #904 → #900 → #886. The same operator flags and Helm block now preserve wake admission, management credential isolation and the optional remote-cluster flag together. Qualify simultaneous TCP/UDP joins after an operator restart. |
| #879 | fix/879-relay-flow-metrics | #905 → master. Measure sustainable throughput, drops, CPU and p95/p99 latency against a direct connection. |
| #880 | feat/880-managed-namespace-monitoring | #908 → combined base of #894 and #905. Includes the #869 network profile and #879 relay instrumentation. The authoritative Palworld server FPS declaration merged in module #60. #926 delivers the corresponding catalog pin. Qualify the signal and missing-signal behavior across two tenants. |
| #881 | feat/881-staged-template-rollouts | #909 → combined base of #907 and #904. Built from fix/870-backup-consistency and test/878-wake-join-burst. Complete #870/#878 and demonstrate restored canary joins, a halted batch and safe rollback. |
| #882 | fix/882-offline-helm | #901 → #900 → #886. The real Argo CD controller runs after the RBAC phase on the same populated installation. It checks sync, prune, issuer and client rotation, waker enforcement and retained claims. |
| #883 + #884 | fix/883-884-mode-scoped-rbac | #900 → #886. One authorization matrix covers the optional permissions, two game identities and upgrade-safe Role naming. |
#917 merged into master. Its browser infrastructure is available to #903, #906 and #911. #910 inherits it through #903. The CI prerequisite tracks recurring hosted browser setup failures identified during qualification, separately from the 18 graded findings.
#922 merged into master and satisfies the image-build prerequisite for #911. The image packaging prerequisite retains the pinned CA upgrade and gives SteamCMD-based builds signed archive fallbacks when the primary Ubuntu hosts are unavailable.
Separate follow-up #921 records the mislabeled SteamCMD base image and the mixed-architecture scheduling risk for five game templates in the audited snapshot. Four in-tree templates still inherit that base after the ARK: Survival Evolved image change. Its implementation is outside this audit stack.
Upstream merged module #59 and module #60. Merge the shared catalog pin in #926 before #907 and #908. Commit 89645fa6 pins f9e55aa1, including both module merges and the ARK: Survival Evolved 2.0.1 legacy-world safeguard. The maintainer also included the Minecraft Java 2.11.0 version picker from module #62. Migration issue · Safeguard PR #64 · Maintainer pin confirmation. The signed OCI release still needs to publish ARK: Survival Ascended 1.0.9, Project Zomboid 1.0.6 and Palworld 1.3.0 before registry consumers can install these versions.
Merge and closure criteria
- A PR description names its parent, child issues, owned files/interfaces and the test that proves the change. Keep each finding’s implementation and verification together. A combined PR closes both linked findings only when both sets of criteria pass.
- Keep tests with their provider. Extend the existing Cilium, backup-lifecycle, audit, placement and published-upgrade fixtures before starting additional harnesses. Retain passing behavior for existing CRs, PVCs and the SQLite single-writer path.
- Close a finding when its acceptance criteria are recorded in its issue. For cluster, game or release drills, include the version, topology or provider, exercised fault, observed result and evidence link. Track platform and companion module releases together where users need both.
Activity
[-]Production readiness audit: remediation tracker and PR execution plan[/-][+]chore(audit): track production readiness findings and PR merge order[/+]Baylem commentedon Nov 11, 2026
Qualification and merge handoff
All 18 child issues have native GitHub closing links. The maintainer merged #888, #889, #899, #903, #905 and #910, closing #873, #868, #871, #875, #879 and #876. The CI prerequisites #917 and #922 also merged, closing #916 and #920. #894 covers #869 and #885, and #900 covers #883 and #884. The issue description lists each review base and its prerequisites.
On upstream
masterate3826d8d, eight of the 19 audit and CI prerequisite PRs had merged. Including the separate catalog pin #926 with the 11 remaining audit PRs, all 66 pairs merged without conflicts. Applying all 12 scoped changes in dependency order produced tree347082765ded031ceda4978e177f0f751f29a4aewith the final catalog pinf9e55aa1. The core Linux Go services, end-to-end suite and PostgreSQL integration tests compiled together. Helm lint passed and the combined production profile rendered 27 NetworkPolicies, two PodMonitors, three PodDisruptionBudgets and one ResourceQuota.4cbb65a8), the admission test verified the placement policy’s specific rejection, and migration waited for the previous volume writer. A 32 MiB OOM raised the kernel max/oom/oom_kill counters from 0/0/0 to 26/1/2. Neighbor Minecraft p95/p99 was 0.2/0.3 ms under pressure and Restic verified in 17.23 s. On #910 (694264df), all eight wake handoffs passed. The local-path and CSI claims retained their original PV/PVC identities and world markers, recovering in 4 s and 5 s after drain. The repeat OOM reached 21/1/2, neighbor p95/p99 was 0.2/2.1 ms and Restic verified in 19.06 s. Resource and admission drill · Drain and repeat OOM drill48887be2. On the retargeted #906 commitddba490b, the same published-release consumer passed the authenticated upgrade in 31.74 s. Retargeted release consumer · Ordinary #906 CI, 110 passing and one skipped. GitOps drill · Registry digests51553ca3), two native API replicas sharing one writable PostgreSQL primary served 172 authenticated requests during the three-node drill. It exercised rolling replica replacement, worker loss, database outage and recovery, and revocation of an existing console stream from the other API replica. Physical recovery onto a different PVC reached WAL LSN0/30018D0in 13 s. The running game retained its GameServer, Pod and PVC identities and its acknowledged save counter. All 9 probes passed, with a maximum observation gap of 2.060 s. Kind HA and PITR · PostgreSQL 17.11 integrationAdvisory-lock transactions explicitly use READ COMMITTED for roles configured with REPEATABLE READ. Established streams recheck Kubernetes grants and game UID. The integration and Kind receipts above exercise those fixes. The signed test correction closes the WebSocket handshake response body. The previously failing API lint and all 111 CI jobs passed on
51553ca3. API lint · Final CI report.The November 11 maintainer review requested changelog entries on all 19 Gameplane PRs and a Kubernetes 1.30 chart minimum on #886. Each release note merges with its own change. For 14 Markdown-only updates, the PR conversations link the previous passing CI, the complete Markdown diff and the canceled repeat run. At 01:35 UTC on November 12, full CI had passed on the code-changing heads of #886, #899, #903, #910, #911 and #917. The earlier Markdown-only cancellations and prior passing runs are recorded in each affected review. CodeRabbit review also identified a legacy game Pod that could roll before drain at first pin and a policy test that could pass on a namespace quota rejection. Both are corrected on #903. Regressions also cover orphan volume writers and unsafe probes in historical snapshots. #910 consumes that reviewed change. Operator review.
Current-head CI and review status are visible on each PR. Protected
masterrequires independent code-owner approval. Upstream published an immutable v0.4.0 release from46b719f6after #866 merged. The earlier failed v0.4.0 tag had pointed to07bee47b. #906 passed the anonymous consumer against the corrected commit and is open for review. Release consumer. Both module #59 and module #60 have merged. Gameplane #926 advances the catalog tof9e55aa1and should merge before #907 and #908. The panel and publisher read this shared gitlink. The associated OCI versions for ARK: Survival Ascended, Project Zomboid and Palworld still require release publication.After the upstream rebase of #886, signed commits
9d4b3442,b80d0ab8and28507b9arestored its admission checks and the seven-file #887 and 25-file #900 reviews. #911 kept the same tested commit and its 70-file comparison. All three full CI reports passed. #886 passed 110 jobs, #887 passed 51 with 12 skipped, and #900 passed 63 with five skipped. The live #887 PostgreSQL upgrade took 92.97 s and verified the Secret reference, credential absence from revision history and fail-closed startup. The #900 RBAC and binding-identity upgrade took 95.01 s on amd64 and 92.20 s on arm64. #886 CI before integration with placement · #887 final CI · #900 final CI.The maintainer merged the safeguard, closing module issue #63, and updated #926 to catalog
f9e55aa1. Its ARK guard and regression are identical to the fully tested module revision. This pin also includes the independently merged Minecraft Java 2.11.0 picker. The exact final catalog also passed its repository validator across all 29 modules (13 clean, 16 with warnings, zero errors), the backup capability test and the Palworld FPS regression. The previous catalog head passed 104 CI jobs with six intentional skips. Maintainer confirmation · Earlier catalog CI · Final catalog CI, 110 of 110 successful on89645fa6. The review thread is resolved. The master ruleset still requires code-owner review. PR author ValgulNecron is the sole listed code owner. Repository CODEOWNERS.The maintainer merged #903 and #910 together via native stack 931. Their common merge commit contains both reviewed heads. #907 initially retargeted to master after #899 and #889 landed. Its arm64 operator test exposed a shared webhook fixture defect that #900 already fixes. #907 first incorporated #900 at
8cff7b87with the same 50-file change. The original BusyBox receiver lost 150 of 300 requests in a controlled test. The signed fix delivered all 300 exactly once. Failure · Webhook rerun. #906 has a 12-file release-consumer diff against its refreshed GitOps and browser base. Placement stack merge · Historical #907 master run · #906 release consumer.The merged ARK: Survival Evolved startup guard detects both the 1.0 saved-data-only mount and the 1.1 installation-root mount before the vendor entrypoint can start a world. At
36f0b8dd, hosted CI passed all 19 volume-layout cases and a preflight against the pinned vendor image. The validator scanned 29 modules with zero errors, and both devcontainer architectures passed. Module PR #64 · Guard validation.On the merged placement commit
e3826d8d, both storage drivers recovered the original worlds and PVC identities in four seconds. Under the 32 MiB tenant limit, the kernel counters reached 41/1/2 from 0/0/0, the neighboring game recorded 0.2/1.6 ms p95/p99 tick times, and the Restic check took 17.445 seconds. Master CI passed 108 jobs with two skips. Live drill · Master CI report.On
8cff7b87, the arm64 operator test passed, as did the backup and offline-recovery drills. The amd64 placement drill independently failed before applying the quota because its Helm release and its disposable tenant shared a namespace. #886 now renders this fixture using the control-plane release namespace and keeps both the admission binding and the quota scoped to the disposable tenant. The overlapping namespace still fails admission validation in Helm. #907 consumes this change at30db243f; its scoped diff remains 50 files, 473 additions and 61 deletions. Passing arm64 webhook test · Placement failure · Shared fix · Current backup CI.