From 7cc1c768e93b818db571a6aa4cb506e3f8b4df67 Mon Sep 17 00:00:00 2001 From: Roger Luethi Date: Mon, 27 Jul 2026 21:13:39 +0200 Subject: [PATCH] kolla: always use the active backup path mariadb_backup_target selects which script kolla-ansible runs for a database backup. Neither of its two values is sound as shipped, so stop deriving it from the load balancer and pin it to "active", which the companion patch described below makes correct everywhere. Why "replica" is broken. kolla-ansible runs mariabackup in a container on mariadb_backup_host and copies the data directory of that same host. The replica script then points mariabackup's SQL control connection somewhere else on purpose: it asks through the VIP which node is serving, and picks the first node that is not that one. Because mariadb_backup_host defaults to the first host of the MariaDB shard group, and both load balancers prefer that same first host, the node the script excludes is normally the very node being copied -- so it does not merely risk selecting a different node, it systematically does. That matters because mariabackup is a physical backup tool: it takes its target LSN over the SQL connection while its redo-copy thread reads the local redo log. Galera replicates at writeset level rather than redo level, so the two LSN sequences are unrelated, and the copy thread waits until the local counter happens to reach a foreign value. Upstream's 2026-08-25 fix pins that control connection to the local node, which makes the exclusion deterministic rather than incidental: the script now always resolves the node being copied, and always picks another. Measured on a 3-node cluster: 30 s pointed at the local node, 17 m 30 s pointed at a sibling 1.5 MB ahead, and unbounded as that offset grows. Why "active" was only accidentally right. It sends mariabackup to the VIP, which works only while the load balancer's chosen node happens to be mariadb_backup_host. That coincidence held with HAProxy. It cannot hold from 2025.2 on, because kolla-ansible removed the internal HAProxy MariaDB frontend there: the VIP can then only be ProxySQL, and ProxySQL announces its own version in the connection handshake (measured: "8.0.11 (ProxySQL)"), which mariabackup rejects outright with "Unsupported server version". Note this is a separate failure from the stall above -- a fast, loud error rather than a hang. Making ProxySQL announce the backend version would clear that handshake, and on a default deployment it would even let "active" work through the VIP: ProxySQL routes to the writer, the writer is the first shard host, and mariadb_backup_host defaults to that same host. That is not an adequate fix. It buys correctness only for as long as those three coincide. A failover moves the writer, and an operator who sets mariadb_backup_host to a non-writer -- the supported way to keep backup load off the node serving traffic -- would have mariabackup copy one node while taking its target LSN from another, reproducing exactly the stall described above. Fixing the announcement is worthwhile in its own right, but as a route to a working backup it trades a loud failure for a latent, configuration-dependent one. So the conditional has nothing worth deciding. Upstream now points mariabackup at the node it backs up, which takes the load balancer out of the control path altogether, and the commits this tag pins for 2025.1, 2025.2 and 2026.1 all carry it -- 2025.2 and 2026.1 are pinned at that very commit. Wherever this value changes behaviour, that fix is present. OSISM forces enable_proxysql off for 2024.1, 2024.2 and 2025.1, so those deploy HAProxy and the conditional already yields "active" there -- pinning it is a no-op. 2024.1 is inert regardless, defining neither mariadb_backup_target nor the replica script. Only 2025.2 and 2026.1 move from "replica" to "active", and both connect to the local node. No image patch is required. The one combination this does not help is ProxySQL enabled by hand on 2024.2, where the template still renders host={{ database_address }} and "active" would meet the handshake described above. That branch is retired upstream and OSISM ships it security fixes only, so it will not gain the local-connect change. Such a deployment has no working backup path either way: "replica" stalls and "active" cannot connect. Upstream reports for the two failure modes: https://bugs.launchpad.net/kolla-ansible/+bug/2091559 (the stall) https://bugs.launchpad.net/kolla-ansible/+bug/2080818 (the handshake) Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi --- all/099-kolla.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/all/099-kolla.yml b/all/099-kolla.yml index 5b28025..e64ae67 100644 --- a/all/099-kolla.yml +++ b/all/099-kolla.yml @@ -92,7 +92,13 @@ enable_mariabackup: "yes" # every release via the enable_proxysql gate. 001 mirrors the 2025.2 literals. mariadb_loadbalancer: "{{ 'proxysql' if enable_proxysql | bool else 'haproxy' }}" mariadb_monitor_user: "{{ 'monitor' if enable_proxysql | bool else 'haproxy' }}" -mariadb_backup_target: "{{ 'active' if mariadb_loadbalancer == 'haproxy' else 'replica' }}" + +# Always use the "active" backup path: "replica" makes mariabackup copy one +# node's data files while taking its target LSN from another, which stalls the +# backup for minutes to days. Only 2025.2 and 2026.1 change value here, and both +# connect the backup to the local node upstream; the releases where OSISM keeps +# HAProxy already resolve to "active". Full rationale in the commit message. +mariadb_backup_target: active # horizon horizon_backend_database: "yes"