Repository navigation
feat(api): qualify active-active PostgreSQL panels and fenced recovery #874
Description
Production readiness audit of Gameplane v0.4.0, commit 07bee47b. Tracking issue and PR merge order: #867.
Finding
The chart enforces one API replica with Recreate. The standard API image omits the postgres build tag. Process-local audit and last-user-manager locks cannot protect two writers. Existing games can keep running while the panel is down.
Work
Deliver and exercise a supported PostgreSQL image/configuration. First prove single-writer failover and point-in-time recovery. For active-active, serialize audit head/prune and last-manager updates in the DB, coordinate migrations and singleton background work, and define cross-replica session/rate-limit behavior before permitting replicas >1 on PostgreSQL. Preserve the SQLite single-writer gate. Treat active-active as an explicit product decision. The recovery drill can ship on the supported single-writer contract.
Accepted deployment model
The approved scope includes active-active Kubernetes-native Gameplane API replicas sharing one writable PostgreSQL primary, with single-writer operation for SQLite and standalone panels. Qualification includes concurrent database invariants, failover of the API and its singleton worker, and fenced point-in-time recovery onto a different volume. Current implementation and evidence
Acceptance criteria
Fence the prior writer, restore to a known WAL boundary, reconcile users/IdP subjects/grants/owners, sessions and management keys against live CR/PVC UIDs, verify positive and denied actions, and record panel/game RPO and RTO. Separately test concurrent writers, rolling upgrade, replica/node/DB loss, audit-chain validity and last-manager safety.
PR ownership and prior work
Split into a recovery drill, database invariants, then HA chart rollout. Reuse H05 configuration and H06 audit internals. Independent recovery/runbook work can start immediately. The singleton was deliberate (#726, #758). Reuse #707 PostgreSQL integration and #839 recovery coverage. TestPanel_OfflineRecovery covers SQLite.
Chart validation · API deployment strategy · User-management lock · PostgreSQL build
Activity
[-][High H07] Qualify PostgreSQL recovery and remove single-writer limits only after HA invariants hold[/-][+]feat(api): verify PostgreSQL recovery and gate multi-replica deployments[/+]Baylem commentedon Nov 11, 2026
The database integration tests passed on PostgreSQL 17 at
42e85dc2in #911. Two API pools connected to the same writable primary. The suite exercised concurrent migration and audit writes, protection of the last user manager, shared authentication limits, session revocation, and worker election after terminating the leader's database connection. A separate test fenced both a pooled connection and new connections read-only. Authenticated reads and readiness returned HTTP 503 until the primary accepted writes again. Process liveness and the original session survived. PostgreSQL build, vet and integration jobThe physical WAL restore, populated-world continuity measurements, and three-node active-active rollout and loss drills have separate Kubernetes checks. Their results are still pending.
Baylem commentedon Nov 11, 2026
The physical recovery and active-active Kubernetes drills both passed at
42e85dc2. The physical restore fenced the original API and PostgreSQL instance and replayed WAL onto a different PVC to the named restore point at LSN0/30165F8. The panel recovered in 12 seconds. The sampled recovery-point age rounded to 0 seconds at one-second timestamp resolution. The game kept the same GameServer, PVC and Pod UIDs, with nine successful progress samples and no lost acknowledged counter. The largest gap between observations was 2.22 seconds. The drill also verified authorization after reapplying the external revocation, invalidating old sessions and reopening the retained world through its original agent identity. Physical restore and HA jobIn a separate three-node kind cluster, two native Gameplane API replicas shared one writable PostgreSQL primary. The Service served 179 authenticated requests during replacement of both replicas. Worker ownership transferred after eviction, and the surviving API continued to accept authenticated writes while the other worker node was unavailable. Both APIs rejected writes during the database outage and recovered against its retained data. These are measurements from the CI fixture. Active-active drill
The follow-up commit
3459d379includes the current wake-admission prerequisite, generic authentication error handling and lint fixes. Its qualification run will verify that exact revision.Baylem commentedon Nov 11, 2026
The standalone PostgreSQL notification path passed on #911 commit
f4fed0f6. CI ran the PostgreSQL-tagged tests uncached against PostgreSQL 17.11. The regression decoded remote Backup objects through client-go and the production poller, sent two distinct failure transitions with long resource versions sharing the same prefix, and replayed the first. The webhook saw exactly two distinct messages, and the database held exactly two claims. The database, standalone management storage, and API runtime packages all passed. PostgreSQL job · RegressionThe same commit passed the three-node Kubernetes qualification. Physical WAL replay onto a separate PVC reached LSN
0/3001BA0. Panel recovery took 14 seconds. Recovery-point age rounded to 0 seconds at one-second resolution. All nine live game probes succeeded, the largest observation gap was 2.10 seconds, and no acknowledged counter was lost. The GameServer, Pod and PVC each retained their UID. In a separate HA cluster, two native API replicas served 172 authenticated requests across replacement of both Pods. The 190.31-second failover test exercised worker handoff, surviving API writes during worker-node loss, rejection during DB outage and service recovery. The populated singleton upgrade and outage drill also passed. Final-head physical recovery and active-active run[-]feat(api): verify PostgreSQL recovery and gate multi-replica deployments[/-][+]feat(api): qualify active-active PostgreSQL panels and fenced recovery[/+]Baylem commentedon Nov 11, 2026
The final qualification on af402c0 passed against PostgreSQL 17.11 and a single-node Kind cluster for singleton upgrade and physical recovery, followed by a fresh three-node Kind cluster for active-active failover. Physical WAL replay to LSN 0/3001C90 restored the panel in 14 seconds. The observed recovery-point age was 0 seconds at one-second resolution. All nine live-game probes succeeded, the maximum observation gap was 2.075 seconds, and the drill retained every acknowledged counter and the original GameServer, Pod and PVC UIDs. The separate active-active rollout served 174 authenticated requests while replacing both API Pods. The worker lock transferred to the surviving API, which also accepted authenticated writes during worker-node loss. The failover drill passed in 194.31 seconds. The populated singleton upgrade and database outage recovery passed in 85.36 seconds. Physical recovery, active-active and singleton drills · PostgreSQL 17.11 integration and standalone notifications
Baylem commentedon Nov 12, 2026
Qualified the current #911 feature head
51553ca3dda638f6144fdf6de385b21c29d0189fagainst PostgreSQL 17.11. The full tagged database, control-plane and API packages passed, including concurrent startup under a REPEATABLE READ default, last-manager protection for a replica waiting on the advisory lock, standalone webhook delivery and worker failover. The corrected API lint job reported zero findings. PostgreSQL integration · API lintThe single-node availability and outage drill passed in 81.48 seconds. Physical recovery replayed archived WAL to LSN
0/30018D0. The panel recovered in 13 seconds. The observed age of the recovery point was zero at one-second resolution. All nine game probes succeeded, the maximum observation gap was 2.060 seconds, and the game retained its CR, Pod and PVC UIDs and acknowledged counters. On a separate three-node Kind cluster, the rollout served 172 authenticated requests while replacing both API Pods. Worker ownership transferred, and the drill verified that removing a collaborator through one replica ended its established live game log stream on the other while the session remained valid. The active-active failover test passed in 193.23 seconds. Kind availability, physical restore and HA drillsAll 111 CI jobs finished successfully on this feature head. After the full run, the target branch alone advanced to
76f384feto include the final #917 prerequisite. The new base produces a clean prospective merge with the same changes in all 70 PR files. Successful CI report