Skip to content

Commit 6398e31

Browse files
committed
docs: address review feedback on the HA guide
- Qualify the failover data-loss statement: async replication does not guarantee zero loss by itself; document synchronous_mode / synchronous_mode_strict as the zero-loss option with its trade-offs - Fix the pkill example ('patroni node1.yml' cannot match the started command line; use 'patroni.*node1.yml') - Replace the shortened patronictl outputs with verbatim captures (cluster id, Receive/Replay LSN columns) and state that outputs are captured verbatim from the verification host - Fix the rejoin wording: the observed behavior is timeline follow, not pg_rewind; document when pg_rewind (use_pg_rewind: true) is required - Add a Production Checklist section: systemd unit, watchdog, HAProxy with /master health check, control-plane TLS, three-node etcd Signed-off-by: zhang1939 <13942782872@163.com>
1 parent cf27543 commit 6398e31

2 files changed

Lines changed: 183 additions & 32 deletions

File tree

‎CN/modules/ROOT/pages/master/high_availability_patroni.adoc‎

Lines changed: 88 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Patroni 是 PostgreSQL 生态事实上的标准 HA 控制器,同样适用于 I
2424
|===
2525

2626
本指南全部内容均在 IvorySQL 5beta1(PostgreSQL 19beta1)+ Patroni 4.1.5 + etcd 3.5 环境上实际验证。
27+
文中命令输出均为验证环境的逐字记录(两节点位于同一主机,故输出中地址为 `localhost`)。
2728

2829
== 前置条件
2930

@@ -151,12 +152,12 @@ patroni /etc/patroni/node2.yml &
151152
[literal]
152153
----
153154
$ patronictl -c /etc/patroni/node1.yml list
154-
+ Cluster: ivory_cluster ----+----+-------------+-----+
155-
| Member | Host | Role | State | TL | Lag |
156-
+--------+----------------+---------+-----------+----+-----+
157-
| node1 | 192.168.1.10:5432 | Leader | running | 1 | |
158-
| node2 | 192.168.1.11:5433 | Replica | streaming | 1 | 0 |
159-
+--------+----------------+---------+-----------+----+-----+
155+
+ Cluster: ivory_cluster (7700535459881935048) -+----+-------------+-----+------------+-----+
156+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
157+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
158+
| n1 | localhost:5432 | Leader | running | 1 | | | | |
159+
| n2 | localhost:5433 | Replica | streaming | 1 | 0/4000118 | 0 | 0/4000118 | 0 |
160+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
160161
----
161162

162163
在主库写入、备库读取:
@@ -198,18 +199,19 @@ $ psql -h 192.168.1.10 -p 5432 -U postgres -d postgres -tAc \
198199
[literal]
199200
----
200201
# 在 node1 主机上
201-
pkill -f 'patroni node1.yml'
202+
pkill -f 'patroni.*node1.yml'
202203
203204
# 约 40 秒后,任意节点执行
204205
$ patronictl -c /etc/patroni/node2.yml list
205-
+ Cluster: ivory_cluster ----+----+-------------+-----+
206-
| Member | Host | Role | State | TL |
207-
+--------+----------------+--------+---------+----+
208-
| node2 | 192.168.1.11:5433 | Leader | running | 2 |
209-
+--------+----------------+--------+---------+----+
206+
+ Cluster: ivory_cluster (7700535459881935048) ---+-------------+-----+------------+-----+
207+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
208+
+--------+----------------+--------+---------+----+-------------+-----+------------+-----+
209+
| n2 | localhost:5433 | Leader | running | 2 | | | | |
210+
+--------+----------------+--------+---------+----+-------------+-----+------------+-----+
210211
----
211212

212-
故障前写入的数据在新主库完整无损,且可立即继续写入:
213+
需要说明:异步复制下故障转移本身并不保证零丢失——旧主库已提交但尚未复制的事务可能丢失。
214+
本次演练没有丢数据,是因为故障发生前写入已停止、已提交事务均已复制;新主库可立即继续写入:
213215

214216
[literal]
215217
----
@@ -218,18 +220,47 @@ $ psql -h 192.168.1.11 -p 5433 -U postgres -d postgres -tAc \
218220
3
219221
----
220222

223+
[NOTE]
224+
====
225+
上面的集群运行在**异步**复制(默认行为,`pg_stat_replication` 中显示 `async`):性能最好,
226+
但故障转移可能丢失最后一批未复制的事务。如需零丢失保证,在 Patroni 配置中启用同步复制:
227+
228+
----
229+
bootstrap:
230+
dcs:
231+
synchronous_mode: true
232+
synchronous_mode_strict: false
233+
----
234+
235+
`synchronous_mode` 使提交等待至少一个同步备库确认;`synchronous_mode_strict` 进一步在
236+
没有任何可用备库时拒绝写入(最高持久性,但备库全挂期间写入可用性为零)。请根据业务
237+
在数据安全与写入可用性之间权衡。
238+
====
239+
221240
=== 节点回归
222241

223-
在故障节点重新启动 Patroni。它检测到更高的时间线,自动回退并重新以备库身份加入:
242+
在故障节点重新启动 Patroni。它检测到更高的时间线,直接跟随新时间线并重新以备库身份加入:
224243

225244
[literal]
226245
----
227246
$ patroni /etc/patroni/node1.yml &
228247
$ patronictl -c /etc/patroni/node1.yml list
229-
| node1 | 192.168.1.10:5432 | Replica | streaming | 2 | 0 |
230-
| node2 | 192.168.1.11:5433 | Leader | running | 2 | |
248+
+ Cluster: ivory_cluster (7700535459881935048) -+----+-------------+-----+------------+-----+
249+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
250+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
251+
| n1 | localhost:5432 | Replica | streaming | 2 | 0/403D688 | 0 | 0/403D688 | 0 |
252+
| n2 | localhost:5433 | Leader | running | 2 | | | | |
253+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
231254
----
232255

256+
[NOTE]
257+
====
258+
上面的回归是直接跟随新时间线,之所以可行是因为故障节点宕机期间没有接受过任何写入。
259+
如果旧主库在故障转移前提交过未复制的事务,其时间线已分叉,Patroni 需要 `pg_rewind`
260+
才能让它重新加入:在节点配置中启用 `use_pg_rewind: true`(要求 `wal_log_hints = on`
261+
或 initdb 时开启数据校验和)。
262+
====
263+
233264
=== 计划内切换
234265

235266
维护场景下不经历故障地转移主库角色:
@@ -271,6 +302,47 @@ patronictl -c /etc/patroni/node1.yml switchover ivory_cluster --leader node1 --c
271302
* 各节点的 PostgreSQL 日志记录提升事件(`redo done at ...`、`selected new timeline`);
272303
* 单节点 etcd 是整个集群的单点——生产环境请部署三节点 etcd。
273304

305+
== 生产环境清单
306+
307+
上文所有守护进程均以 shell 后台进程方式运行,便于学习验证,但生产环境不应如此:
308+
309+
* *守护每一个进程。* 将 etcd、Patroni(以及可选的本机 HAProxy)注册为 systemd 服务,
310+
崩溃或重启后自动拉起:
311+
312+
----
313+
# /etc/systemd/system/patroni@.service
314+
[Unit]
315+
Description=Patroni IvorySQL %i
316+
After=network.target etcd.service
317+
318+
[Service]
319+
User=postgres
320+
ExecStart=/usr/local/bin/patroni /etc/patroni/%i.yml
321+
Restart=always
322+
323+
[Install]
324+
WantedBy=multi-user.target
325+
----
326+
327+
* *启用 Patroni watchdog。* 配置 `watchdog: mode: required`,防止失去仲裁的节点继续以
328+
主库身份提供服务。
329+
* *前置负载均衡。* HAProxy 配合 Patroni REST 健康检查,给客户端一个永远指向当前主库
330+
的单一入口:
331+
332+
----
333+
listen ivorysql_primary
334+
bind *:5000
335+
option httpchk GET /master
336+
http-check expect status 200
337+
default-server inter 3s fall 3 rise 2
338+
server node1 192.168.1.10:5432 check port 8008
339+
server node2 192.168.1.11:5433 check port 8009
340+
----
341+
342+
* *保护控制面。* 共享网络中,Patroni REST API 与 etcd 应启用 TLS 与认证
343+
(`restapi: verify_client`、`etcd3: cacert/cert/key`)。
344+
* *etcd 部署三节点。* 单节点 etcd 本身就是单点。
345+
274346
== 常见问题
275347

276348
[cols="1,2"]

‎EN/modules/ROOT/pages/master/high_availability_patroni.adoc‎

Lines changed: 95 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ Patroni is the de-facto standard HA controller for PostgreSQL-compatible databas
2424
|===
2525

2626
Everything in this guide was verified on IvorySQL 5beta1 (PostgreSQL 19beta1) with Patroni 4.1.5 and etcd 3.5.
27+
All command outputs below are captured verbatim from the verification environment (both nodes ran
28+
on a single host, hence the `localhost` addresses in the outputs).
2729

2830
== Prerequisites
2931

@@ -151,12 +153,12 @@ patroni /etc/patroni/node2.yml &
151153
[literal]
152154
----
153155
$ patronictl -c /etc/patroni/node1.yml list
154-
+ Cluster: ivory_cluster ----+----+-------------+-----+
155-
| Member | Host | Role | State | TL | Lag |
156-
+--------+----------------+---------+-----------+----+-----+
157-
| node1 | 192.168.1.10:5432 | Leader | running | 1 | |
158-
| node2 | 192.168.1.11:5433 | Replica | streaming | 1 | 0 |
159-
+--------+----------------+---------+-----------+----+-----+
156+
+ Cluster: ivory_cluster (7700535459881935048) -+----+-------------+-----+------------+-----+
157+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
158+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
159+
| n1 | localhost:5432 | Leader | running | 1 | | | | |
160+
| n2 | localhost:5433 | Replica | streaming | 1 | 0/4000118 | 0 | 0/4000118 | 0 |
161+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
160162
----
161163

162164
Write on the leader and read from the replica:
@@ -198,18 +200,21 @@ Stop the Patroni process on the leader (simulating a host failure). The survivin
198200
[literal]
199201
----
200202
# on node1's host
201-
pkill -f 'patroni node1.yml'
203+
pkill -f 'patroni.*node1.yml'
202204
203205
# ~40 seconds later, on any node
204206
$ patronictl -c /etc/patroni/node2.yml list
205-
+ Cluster: ivory_cluster ----+----+-------------+-----+
206-
| Member | Host | Role | State | TL |
207-
+--------+----------------+--------+---------+----+
208-
| node2 | 192.168.1.11:5433 | Leader | running | 2 |
209-
+--------+----------------+--------+---------+----+
207+
+ Cluster: ivory_cluster (7700535459881935048) ---+-------------+-----+------------+-----+
208+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
209+
+--------+----------------+--------+---------+----+-------------+-----+------------+-----+
210+
| n2 | localhost:5433 | Leader | running | 2 | | | | |
211+
+--------+----------------+--------+---------+----+-------------+-----+------------+-----+
210212
----
211213

212-
Data written before the failure is intact on the new leader, which accepts writes immediately:
214+
With asynchronous replication a failover does not by itself guarantee zero data loss: transactions
215+
committed on the old primary but not yet replicated can be lost. In this drill nothing was lost,
216+
because writes had already ceased and all committed transactions were replicated; the new leader
217+
accepts writes immediately:
213218

214219
[literal]
215220
----
@@ -218,18 +223,50 @@ $ psql -h 192.168.1.11 -p 5433 -U postgres -d postgres -tAc \
218223
3
219224
----
220225

226+
[NOTE]
227+
====
228+
The cluster above runs with *asynchronous* replication (the default, as shown by `async` in
229+
`pg_stat_replication`): best performance, but a failover can lose the last unreplicated
230+
transactions. If zero data loss is required, enable synchronous replication:
231+
232+
----
233+
bootstrap:
234+
dcs:
235+
synchronous_mode: true
236+
synchronous_mode_strict: false
237+
----
238+
239+
`synchronous_mode` makes commits wait for at least one synchronous standby;
240+
`synchronous_mode_strict` additionally refuses writes when no standby is available (maximum
241+
durability, zero write availability during a total standby failure). Weigh data safety against
242+
write availability for your workload.
243+
====
244+
221245
=== Node Rejoin
222246

223-
Restart Patroni on the failed node. It notices the higher timeline, rewinds itself and rejoins automatically as a replica:
247+
Restart Patroni on the failed node. It notices the higher timeline, follows it and rejoins automatically as a replica:
224248

225249
[literal]
226250
----
227251
$ patroni /etc/patroni/node1.yml &
228252
$ patronictl -c /etc/patroni/node1.yml list
229-
| node1 | 192.168.1.10:5432 | Replica | streaming | 2 | 0 |
230-
| node2 | 192.168.1.11:5433 | Leader | running | 2 | |
253+
+ Cluster: ivory_cluster (7700535459881935048) -+----+-------------+-----+------------+-----+
254+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
255+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
256+
| n1 | localhost:5432 | Replica | streaming | 2 | 0/403D688 | 0 | 0/403D688 | 0 |
257+
| n2 | localhost:5433 | Leader | running | 2 | | | | |
258+
+--------+----------------+---------+-----------+----+-------------+-----+------------+-----+
231259
----
232260

261+
[NOTE]
262+
====
263+
The rejoin above is a plain timeline follow, which works because the failed node had not accepted
264+
any writes of its own while it was down. If the old primary committed transactions that were never
265+
replicated before the failover, its timeline has diverged and Patroni needs `pg_rewind` to rejoin
266+
it: enable `use_pg_rewind: true` in the node configuration (requires `wal_log_hints = on` or data
267+
checksums enabled at initdb time).
268+
====
269+
233270
=== Planned Switchover
234271

235272
For maintenance, move the leader role without any failure:
@@ -271,6 +308,48 @@ Monitoring essentials:
271308
* PostgreSQL logs of each node record promotion events (`redo done at ...`, `selected new timeline`).
272309
* etcd health is a cluster-wide single point of failure in single-node setups — use three etcd nodes in production.
273310

311+
== Production Checklist
312+
313+
The deployment above runs all daemons as background shell processes, which is convenient for
314+
learning but is not how a production HA cluster should run:
315+
316+
* *Supervise every daemon.* Run etcd, Patroni (and optionally a local HAProxy) as systemd services
317+
so they come back after crashes or reboots:
318+
319+
----
320+
# /etc/systemd/system/patroni@.service
321+
[Unit]
322+
Description=Patroni IvorySQL %i
323+
After=network.target etcd.service
324+
325+
[Service]
326+
User=postgres
327+
ExecStart=/usr/local/bin/patroni /etc/patroni/%i.yml
328+
Restart=always
329+
330+
[Install]
331+
WantedBy=multi-user.target
332+
----
333+
334+
* *Enable the Patroni watchdog.* Set `watchdog: mode: required` so a node that loses quorum cannot
335+
keep serving as the primary.
336+
* *Put a load balancer in front.* HAProxy with the Patroni REST health checks gives clients a
337+
single endpoint that always routes to the current primary:
338+
339+
----
340+
listen ivorysql_primary
341+
bind *:5000
342+
option httpchk GET /master
343+
http-check expect status 200
344+
default-server inter 3s fall 3 rise 2
345+
server node1 192.168.1.10:5432 check port 8008
346+
server node2 192.168.1.11:5433 check port 8009
347+
----
348+
349+
* *Secure the control plane.* The Patroni REST API and etcd should use TLS and authentication
350+
(`restapi: verify_client`, `etcd3: cacert/cert/key`) in shared networks.
351+
* *Use three etcd nodes.* A single-node etcd is itself a single point of failure.
352+
274353
== Troubleshooting
275354

276355
[cols="1,2"]

0 commit comments

Comments
 (0)