@@ -24,6 +24,8 @@ Patroni is the de-facto standard HA controller for PostgreSQL-compatible databas
2424|===
2525
2626Everything in this guide was verified on IvorySQL 5beta1 (PostgreSQL 19beta1) with Patroni 4.1.5 and etcd 3.5.
27+ All command outputs below are captured verbatim from the verification environment (both nodes ran
28+ on a single host, hence the `localhost` addresses in the outputs).
2729
2830== Prerequisites
2931
@@ -151,12 +153,12 @@ patroni /etc/patroni/node2.yml &
151153[literal]
152154----
153155$ patronictl -c /etc/patroni/node1.yml list
154- + Cluster: ivory_cluster - ---+----+- ------------+-----+
155- | Member | Host | Role | State | TL | Lag |
156- +--------+----------------+---------+-----------+----+-----+
157- | node1 | 192.168.1.10 :5432 | Leader | running | 1 | |
158- | node2 | 192.168.1.11 :5433 | Replica | streaming | 1 | 0 |
159- +--------+----------------+---------+-----------+----+-----+
156+ + Cluster: ivory_cluster (7700535459881935048) -+ ---- +-------------+-----+ ------------+-----+
157+ | Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
158+ +--------+----------------+---------+-----------+----+-------------+-----+------------+----- +
159+ | n1 | localhost :5432 | Leader | running | 1 | | | | |
160+ | n2 | localhost :5433 | Replica | streaming | 1 | 0/4000118 | 0 | 0/4000118 | 0 |
161+ +--------+----------------+---------+-----------+----+-------------+-----+------------+----- +
160162----
161163
162164Write on the leader and read from the replica:
@@ -198,18 +200,21 @@ Stop the Patroni process on the leader (simulating a host failure). The survivin
198200[literal]
199201----
200202# on node1's host
201- pkill -f 'patroni node1.yml'
203+ pkill -f 'patroni.* node1.yml'
202204
203205# ~40 seconds later, on any node
204206$ patronictl -c /etc/patroni/node2.yml list
205- + Cluster: ivory_cluster ---- +----+- ------------+-----+
206- | Member | Host | Role | State | TL |
207- +--------+----------------+--------+---------+----+
208- | node2 | 192.168.1.11 :5433 | Leader | running | 2 |
209- +--------+----------------+--------+---------+----+
207+ + Cluster: ivory_cluster (7700535459881935048) ---+-------------+-----+ ------------+-----+
208+ | Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
209+ +--------+----------------+--------+---------+----+-------------+-----+------------+-----+
210+ | n2 | localhost :5433 | Leader | running | 2 | | | | |
211+ +--------+----------------+--------+---------+----+-------------+-----+------------+-----+
210212----
211213
212- Data written before the failure is intact on the new leader, which accepts writes immediately:
214+ With asynchronous replication a failover does not by itself guarantee zero data loss: transactions
215+ committed on the old primary but not yet replicated can be lost. In this drill nothing was lost,
216+ because writes had already ceased and all committed transactions were replicated; the new leader
217+ accepts writes immediately:
213218
214219[literal]
215220----
@@ -218,18 +223,50 @@ $ psql -h 192.168.1.11 -p 5433 -U postgres -d postgres -tAc \
218223 3
219224----
220225
226+ [NOTE]
227+ ====
228+ The cluster above runs with *asynchronous* replication (the default, as shown by `async` in
229+ `pg_stat_replication`): best performance, but a failover can lose the last unreplicated
230+ transactions. If zero data loss is required, enable synchronous replication:
231+
232+ ----
233+ bootstrap:
234+ dcs:
235+ synchronous_mode: true
236+ synchronous_mode_strict: false
237+ ----
238+
239+ `synchronous_mode` makes commits wait for at least one synchronous standby;
240+ `synchronous_mode_strict` additionally refuses writes when no standby is available (maximum
241+ durability, zero write availability during a total standby failure). Weigh data safety against
242+ write availability for your workload.
243+ ====
244+
221245=== Node Rejoin
222246
223- Restart Patroni on the failed node. It notices the higher timeline, rewinds itself and rejoins automatically as a replica:
247+ Restart Patroni on the failed node. It notices the higher timeline, follows it and rejoins automatically as a replica:
224248
225249[literal]
226250----
227251$ patroni /etc/patroni/node1.yml &
228252$ patronictl -c /etc/patroni/node1.yml list
229- | node1 | 192.168.1.10:5432 | Replica | streaming | 2 | 0 |
230- | node2 | 192.168.1.11:5433 | Leader | running | 2 | |
253+ + Cluster: ivory_cluster (7700535459881935048) -+----+-------------+-----+------------+-----+
254+ | Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
255+ +--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
256+ | n1 | localhost:5432 | Replica | streaming | 2 | 0/403D688 | 0 | 0/403D688 | 0 |
257+ | n2 | localhost:5433 | Leader | running | 2 | | | | |
258+ +--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
231259----
232260
261+ [NOTE]
262+ ====
263+ The rejoin above is a plain timeline follow, which works because the failed node had not accepted
264+ any writes of its own while it was down. If the old primary committed transactions that were never
265+ replicated before the failover, its timeline has diverged and Patroni needs `pg_rewind` to rejoin
266+ it: enable `use_pg_rewind: true` in the node configuration (requires `wal_log_hints = on` or data
267+ checksums enabled at initdb time).
268+ ====
269+
233270=== Planned Switchover
234271
235272For maintenance, move the leader role without any failure:
@@ -271,6 +308,48 @@ Monitoring essentials:
271308* PostgreSQL logs of each node record promotion events (`redo done at ...`, `selected new timeline`).
272309* etcd health is a cluster-wide single point of failure in single-node setups — use three etcd nodes in production.
273310
311+ == Production Checklist
312+
313+ The deployment above runs all daemons as background shell processes, which is convenient for
314+ learning but is not how a production HA cluster should run:
315+
316+ * *Supervise every daemon.* Run etcd, Patroni (and optionally a local HAProxy) as systemd services
317+ so they come back after crashes or reboots:
318+
319+ ----
320+ # /etc/systemd/system/patroni@.service
321+ [Unit]
322+ Description=Patroni IvorySQL %i
323+ After=network.target etcd.service
324+
325+ [Service]
326+ User=postgres
327+ ExecStart=/usr/local/bin/patroni /etc/patroni/%i.yml
328+ Restart=always
329+
330+ [Install]
331+ WantedBy=multi-user.target
332+ ----
333+
334+ * *Enable the Patroni watchdog.* Set `watchdog: mode: required` so a node that loses quorum cannot
335+ keep serving as the primary.
336+ * *Put a load balancer in front.* HAProxy with the Patroni REST health checks gives clients a
337+ single endpoint that always routes to the current primary:
338+
339+ ----
340+ listen ivorysql_primary
341+ bind *:5000
342+ option httpchk GET /master
343+ http-check expect status 200
344+ default-server inter 3s fall 3 rise 2
345+ server node1 192.168.1.10:5432 check port 8008
346+ server node2 192.168.1.11:5433 check port 8009
347+ ----
348+
349+ * *Secure the control plane.* The Patroni REST API and etcd should use TLS and authentication
350+ (`restapi: verify_client`, `etcd3: cacert/cert/key`) in shared networks.
351+ * *Use three etcd nodes.* A single-node etcd is itself a single point of failure.
352+
274353== Troubleshooting
275354
276355[cols="1,2"]
0 commit comments