[Baylem]

Closes #874. Part of #867.

This PR implements the approved active-active option for Kubernetes-native APIs on external PostgreSQL. SQLite and standalone retain the existing single-replica guard. #888, #917 and #922 have merged. Merge #887 and #904 first. The combined base includes #900 and #886 through #904. The shared Deployment flags and suite selection have been reconciled. Retarget this PR to master and rerun CI after those parents land.

The release API image contains both SQL drivers. The native Helm path requires an explicit HA setting, a Secret-backed DSN, and two or more API replicas on separate hosts. Existing installations first roll to an HA-capable singleton image. Migrations and their initial seed commit under one schema-scoped transaction lock. Another transaction lock protects account, role, binding and OIDC mutations through commit. PostgreSQL serializes audit append and prune, and verification reads a consistent snapshot.

The fenced recovery drill verifies a physical base backup, replays to a named WAL restore point, reapplies the recorded IdP revocation, and invalidates restored bearer credentials before the panel can serve traffic. The drill checks a permitted and a denied operation, live GameServer ownership and UIDs, the PVC, agent credential, audit chain and a retained world marker. CI records the target LSN, observed recovery-point age and end-to-end recovery time.

GitHub Actions qualifies the combined image and chart. The PostgreSQL integration lane exercises simultaneous startup, last-manager serialization, concurrent audit writes, shared login budgets, revocation and worker ownership after backend loss. The three-node kind lane checks direct writes to both API replicas, rolling upgrade through the Service, loss of the active worker replica, node loss, database outage and recovery, and the pinned legacy image rollback guard. A named PostgreSQL HA startup flag prevents a legacy binary from opening the database as an additional writer. The bootstrap-admin command now commits its account and grant atomically.

A live PostgreSQL test fences an already pooled connection and every new connection, then checks that authenticated reads and both readiness routes fail closed until the writable endpoint returns.

The earlier qualification on af402c0 passed against PostgreSQL 17.11 and a single-node Kind cluster for singleton upgrade and physical recovery, followed by a fresh three-node Kind cluster for active-active failover. Physical WAL replay to LSN 0/3001C90 restored the panel in 14 seconds. The observed recovery-point age was 0 seconds at one-second resolution. All nine live-game probes succeeded, the maximum observation gap was 2.075 seconds, and the drill retained every acknowledged counter and the original GameServer, Pod and PVC UIDs. The separate active-active rollout served 174 authenticated requests while replacing both API Pods. The worker lock transferred to the surviving API, which also accepted authenticated writes during worker-node loss. The failover drill passed in 194.31 seconds. The populated singleton upgrade and database outage recovery passed in 85.36 seconds. Physical recovery, active-active and singleton drills · PostgreSQL 17.11 integration and standalone notifications

The follow-up commit 3459d379 includes the current wake-admission prerequisite, generic authentication error handling and lint fixes. At that revision the live PostgreSQL tests passed, and both ordinary API architectures reached 83.8% coverage. PostgreSQL integration · amd64 API · arm64 API

Commit f4fed0f6 restores standalone PostgreSQL notification delivery. Remote polling had dropped the Kubernetes resource version needed to claim a transition for each sink. A new PostgreSQL regression sends two distinct transitions and replays one through the production poller and a local HTTP webhook, checking for exactly two dispatches. The regression passed against PostgreSQL 17.11 on that commit, including exactly two accepted webhook deliveries and two database claims. PostgreSQL result Verification run at f4fed0f6

The earlier combined base e83de60 included the final #917 browser setup and #922 game-image archive fallback. Previous feature head af402c0 retained the same 64-file PostgreSQL patch as f4fed0f. The intermediate 8de89f1 passed the real PostgreSQL 17.11 integration. PostgreSQL at 8de89f1e · Earlier PostgreSQL and Kind qualification · Earlier game-image builds
Commit aa89dadf5ac571c9dfda8b8c799a304d2c2350be added two protections. PostgreSQL migrations and account changes explicitly use READ COMMITTED so a waiting replica sees the previous holder’s commit even under a REPEATABLE READ role default. Established server streams on PostgreSQL periodically recheck the Kubernetes owner or collaborator grant against the original GameServer UID. The real PostgreSQL regression observes the blocked second transaction, commits the first demotion, then checks the second is denied and one manager remains. The live three-node test removes a collaborator through one replica while the collaborator is reading the game's Pod logs through the other. Qualification run for both corrections
Current head 51553ca3dda638f6144fdf6de385b21c29d0189f targets stack/874-postgres-prerequisites. All 111 CI jobs passed against integration base 7d3e0109. After qualification, the target branch advanced first to 76f384fe for #917 and then to 02cd120f for the repaired #886, #887 and #900 ancestry. The prospective merge is clean. All 70 native PR files retain their original additions and removals, and the GitHub comparison with the current target matches the native review. Successful CI report. It closes the handshake response in the stream regression test and includes the requested upgrade and recovery changelog entry. The prerequisite branch incorporates the updated Kubernetes 1.30 minimum, DSN configuration, browser CI and image-build changes.
The exact feature head passed PostgreSQL 17.11 integration, including the advisory-lock handoff, standalone notification delivery and cross-replica worker recovery. API lint reported zero findings. The single-node upgrade and outage drill completed in 81.48 seconds. The physical restore replayed WAL to LSN 0/30018D0 and the panel recovered in 13 seconds. All nine game probes succeeded with the original UIDs and acknowledged counters. On a new three-node Kind cluster, the two API replicas served 172 authenticated rollout requests, transferred worker ownership, and closed an active stream after a collaborator’s grant was revoked through the other replica. The HA drill passed in 193.23 seconds. PostgreSQL integration · API lint · Kind drills · Full CI run

[coderabbitai]

Pre-merge checks: 1 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Title check		This assessment has not completed. The check will be retried on the next pre-merge check run.

Passed checks: 1 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[github-actions]

CI Report - 51553ca

run 43111916106 - attempt 1

0 failed - 33 passed - 0 skipped - 88m56s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module	Measured	Gate	Margin
web	L94.03 S92.14 B86.51 F90.19	L92 S92 B82 F76	+0.14

Changed

Area	Files	+/-
api/	48	+1930/-388
test/	6	+692/-4
docs/	3	+128/-25
deploy/	3	+122/-20
charts/	6	+92/-19
.github/	2	+39/-19
containers/	1	+0/-9
changelog/	1	+5/-0

Slowest jobs

Job	Duration
e2e telemetry / arm64 (kind)	23m6s
e2e telemetry / amd64 (kind)	22m38s
e2e postgres / amd64 (kind)	20m15s
e2e restricted-workloads / arm64 (kind)	15m49s
e2e operator / arm64 (kind)	12m50s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, bot-fast, management-placement, multicluster, operator, optional-components, postgres-availability, postgres-ha, postgres-pitr, ratelimit, restricted-workloads, telemetry, upgrade

[ValgulNecron]

Hey, could you please add changelog entry?

[Baylem]

Added the PostgreSQL HA and recovery changelog entry. It covers the two-stage upgrade to multiple native API replicas, the SQLite and standalone singleton requirement, and separate storage for the PostgreSQL base backup and archived WAL.

The same update fixes the failed API lint check by closing the WebSocket handshake response in the grant-revocation regression. Full Linux API lint passed locally with the same Go and golangci-lint versions and build tags as CI. Hosted checks are running on 51553ca3. Current CI