[proposal] High availability strategy for Redis and RabbitMQ (investigation)

[henzigo]

Context

Redis and RabbitMQ run single-replica. A node failure means losing the cache and the
message broker until rescheduling; a drain means a full outage of both. Redis persistence is
intentionally disabled (appendonly no, save "") but that decision is undocumented.

Proposal (investigation, decision record as output)

Primary direction: adopt existing, maintained charts/operators instead of extending our own
templates. The helmfile composition makes this natural — a third-party chart is just another
release in helmfile.yaml.gotmpl (like shopsys-infra today), values stay in our
environments layering, and we stop maintaining Redis/RabbitMQ manifests ourselves.

Candidates to evaluate:

RabbitMQ

• RabbitMQ Cluster Operator
  (official, free) — cluster CRD, quorum queues, rolling upgrades; needs the operator as a
  cluster-wide prerequisite
• Bitnami rabbitmq chart — ⚠️ see the Bitnami caveat below
• keep our StatefulSet only if the decision is "single replica is fine"

Redis

• Bitnami redis chart (master-replica + Sentinel) — ⚠️ Bitnami caveat below
• OT-ContainerKit Redis Operator or
  Spotahome redis-operator
• keep our Deployment if the decision is "ephemeral single-replica cache is fine"
  (then just document it — cheapest correct outcome)

⚠️ Bitnami caveat (2025): Bitnami moved its free public images to the unmaintained
bitnamilegacy registry; the charts default to images that now require a Bitnami Secure
Images subscription. Evaluating Bitnami charts must include the image-source question
(subscription, legacy tags, or overriding with upstream redis/rabbitmq images — which the
charts support but which loses part of their value).

Evaluation criteria

• migration path from today's installs: resource names/labels differ (redis,
  rabbitmq, PVC rabbitmq-data-rabbitmq-0) → RabbitMQ data migration or blue/green
  switchover needed; Service names/DSNs consumed by the app must stay stable
  (redis:6379, rabbitmq:5672) or be remapped via values
• operational cost: operator lifecycle vs chart-only; upgrades; monitoring integration
  (we already scrape exporters — [observability] ServiceMonitors, logging annotations and deploy metadata #39)
• parity constraints: management ingress + IP whitelist, prometheus-exporter labels,
  the deploy-order guarantee (infra release ready before migration hook) must hold for
  whatever replaces shopsys-infra
• does the shop actually need HA here? (failure-mode analysis first — the outcome may be
  "document and accept single replica", which keeps our small templates and costs nothing)

Deliverable: decision record in docs/ + follow-up implementation issues per outcome.

Tasks

• Failure-mode analysis (what breaks for the shop when redis/rabbitmq are down N minutes)
• Evaluate RabbitMQ Cluster Operator and a Redis chart/operator against the criteria above
  (incl. the Bitnami image-source question)
• Migration/switchover plan sketch for existing installs
• Decision record + follow-ups

[henzigo]

Initial assessment

Feasibility assessment for the HA strategy investigation (phase 1). Facts verified against upstream projects as of August 2026. Current state in this repo: charts/shopsys-infra runs Redis as a 1-replica Deployment (appendonly no, save "" — intentionally ephemeral, ~2.5Gi memory) and RabbitMQ as a 1-replica StatefulSet (rabbitmq:4.1-management-alpine, 1Gi PVC on nfs-client, requests only 20m CPU). The app consumes fixed DSNs redis:6379 / rabbitmq:5672, and shopsys-app needs: shopsys-infra with wait: true for the migration-hook ordering.

1. RabbitMQ — Cluster Operator

Status/maturity: actively maintained by the RabbitMQ core team — latest release v2.22.4 (2026-08-11), with monthly releases through 2026. Provides the RabbitmqCluster CRD; official, free, Apache-2.0.

Installation model: the operator is a cluster-wide singleton prerequisite (namespace rabbitmq-system), installed via kubectl apply -f .../cluster-operator.yml or the krew plugin (install docs). There is no official Helm chart; the Bitnami rabbitmq-cluster-operator chart exists but is caught in the Bitnami Secure Images situation (see §2), and a small community chart exists on ArtifactHub. Practical consequence for this repo: the operator does not fit cleanly into per-project helmfile.yaml.gotmpl releases — it becomes a documented cluster prerequisite (like ingress-nginx/cert-manager today), while the per-project RabbitmqCluster CR would be a tiny chart or raw-manifest release replacing the rabbitmq.* half of shopsys-infra.

Parity mapping (using-operator docs):

Concern	Today (shopsys-infra)	With operator (CR named rabbitmq)
DSN rabbitmq:5672	headless Service rabbitmq	Service rabbitmq (ClusterIP), ports 5672/15672/15692 — DSN unchanged ✅
Credentials	Secret rabbitmq-credentials, keys user/password, values from RABBITMQ_DEFAULT_USER/PASS env interpolation	operator generates Secret rabbitmq-default-user, keys username/password — different name, different keys, random values. Anything consuming rabbitmq-credentials (app config, CI) needs remapping, or credentials must be imported via definitions/additionalConfig ⚠️
Management ingress + IP whitelist	our ingress-rabbitmq.yaml → Service rabbitmq:15672	keep our Ingress template pointing at the operator's Service — works as-is ✅
prometheus-exporter: 'true' Service label	on our Service	operator owns the Service; extra labels via spec.override.service (full override support exists), or switch to ServiceMonitor (#39) ✅
Deploy order (infra before migration hook)	needs: + wait: true	wait: true on a CR release waits for the CR object, not for the cluster to be ready — needs a readiness gate (e.g. kubectl wait --for=condition=AllReplicasReady rabbitmqcluster/rabbitmq as a presync/hook) ⚠️
Storage	1Gi PVC, nfs-client	PVC per node, default 10Gi, storageClassName configurable. NFS is explicitly discouraged for RabbitMQ quorum queues (Raft WAL on NFS is a known problem) — 3-node HA realistically wants block storage ⚠️

Actual HA requires more than the operator: classic queue mirroring was removed in RabbitMQ 4.0 — quorum queues are the only replicated queue type (release notes). That means: 3 nodes (odd counts required), and the application must declare its queues with x-queue-type: quorum — Symfony Messenger supports this via transport queue arguments, but it is an app-level change and existing classic queues cannot be converted in place (delete + redeclare, or migrate). Operator defaults are ~1 CPU / 2Gi per node → a 3-node cluster is a step change from today's 20m-CPU single pod.

Migration path: resource names differ (StatefulSet rabbitmq vs operator-managed rabbitmq-server, PVC rabbitmq-data-rabbitmq-0 vs persistence-rabbitmq-server-N), so this is a replace, not an adopt. During a maintenance window: stop producers/consumers, drain queues to empty (normal state for a healthy broker), delete old release, deploy CR, re-create users/vhost. Message loss is acceptable only if queues are drained first — for an e-shop, in-flight order/email messages must be drained or shoveled, not dropped. A blue/green switchover (new CR under a different name, then Service/DSN flip) avoids the window but adds steps.

2. Redis

Bitnami charts (2026 state): effectively off the table. Since 2025-08-28 Bitnami's versioned free images moved to the unmaintained bitnamilegacy registry; the free tier is ~44 dev-only images with only :latest tags; maintained/pinned images require a Bitnami Secure Images subscription (bitnami/charts#35164, The New Stack). The charts do accept upstream redis image overrides, but then you're maintaining an image/chart compatibility matrix yourself (sidecar scripts in the chart assume Bitnami image layout) — the "stop maintaining manifests" benefit largely evaporates.

Operators: Spotahome redis-operator is archived (last push July 2024) — excluded. OT-ContainerKit redis-operator is actively maintained (v0.26.0, 2026-07-15, releases every ~2 months), has a Helm chart, and provides Redis/RedisReplication/RedisSentinel/RedisCluster CRDs — but it's pre-1.0, defaults to its own quay.io/opstree/redis* images, and is another cluster-wide operator to run.

Is Sentinel worth it for an intentionally ephemeral cache? Almost certainly not:

• The data is disposable by design (appendonly no, documented in values.yaml). Failover preserves nothing valuable — a fresh empty Redis is nearly as good as a promoted replica, since the replica's dataset is just a warm cache.
• Sentinel eliminates only the rescheduling gap (roughly 30 s–a few min for a Deployment pod to land elsewhere; longer only if a node dies uncleanly and taint-eviction timeouts apply). The cost: 2–3× the 2.5Gi memory footprint plus 3 sentinel pods, an operator to run, and client changes.
• Client-side implication (open question for the app team): Sentinel is not transparent — clients must speak the Sentinel protocol to discover the master. Symfony's RedisAdapter supports it (redis_sentinel=<master> DSN option, Predis or phpredis ≥5), with known quirks in long-running phpredis processes (symfony#46600). Whether Shopsys's Redis wiring (snc/redis-bundle config, session handler, storefront cache) passes Sentinel DSNs through everywhere is unverified and must be answered by the app team before Sentinel is even a candidate.

The proportionate alternative: keep the Deployment, make rescheduling fast and eviction-safe — see §3.

3. The "document and accept" option

What a single-replica outage actually costs, per component:

Redis down N minutes: Shopsys uses Redis for cache + sessions. Symptoms: users logged out / carts tied to sessions degraded, page latency spikes while the app hits PostgreSQL/Elasticsearch uncached; after recovery, a thundering-herd cache rebuild (minutes of elevated DB load). Unpleasant, self-healing, no data loss by design.

RabbitMQ down N minutes: producers (order placement, email dispatch — depends on how the app publishes) either fail or buffer; consumers idle; queued messages on the PVC survive restart (classic durable queues + NFS PVC), so this is delayed processing, not loss — unless the PVC/node is destroyed. Backlog drains after restart. The failure that actually loses messages (PVC corruption/loss) is not fixed by 3 replicas of classic queues anyway (they're non-replicated in 4.x).

Cheap mitigations, all without HA and without new dependencies:

1. PDB + topologySpreadConstraints ([availability] PodDisruptionBudgets and topologySpreadConstraints #8): protects against voluntary disruption (node drains during cluster upgrades — the most common real outage cause); combined with priorityClassName it minimizes the gap.
2. priorityClass: ensures redis/rabbitmq reschedule ahead of app pods under pressure — the values plumbing already exists in both components, so it's one values line per environment.
3. Fast, non-NFS storage for RabbitMQ where available: shrinks restart/recovery time and removes the NFS-corruption risk — a values-only change (rabbitmq.persistence.storageClass).
4. Tuned probes/short terminationGracePeriod so liveness recovery is quick.
5. Document the ephemerality decision in docs/ (currently undocumented) plus a short runbook: what an outage looks like, expected self-recovery time, when to escalate.

4. Decision matrix

Option	Availability gain	Operational cost	Migration risk	App changes	Resource delta
A. Document & accept + cheap mitigations (PDB, priorityClass, storage)	Drains no longer cause outages; unplanned gap shrinks to rescheduling time	none (keeps our ~9 small templates)	none	none	~0
B. RabbitMQ Cluster Operator, single-node CR	Rolling upgrades w/o manual work; same SPOF as today	operator lifecycle (cluster-wide), CRD upgrades, readiness-gate rework	medium (rename secret/PVC, drain window, hook ordering)	secret name/keys remap	~+1 CPU/+2Gi (operator + defaults; tunable)
C. RabbitMQ Cluster Operator, 3-node quorum	survives node loss with no broker downtime	as B + quorum-queue ops knowledge, block storage requirement	medium-high (queue-type migration, NFS unsuitable)	x-queue-type: quorum in Messenger config + secret remap	~+2–3 CPU/+4–6Gi
D. Redis via OT operator + Sentinel	cache survives node loss warm	pre-1.0 operator, own image defaults, more moving parts than the thing it protects	medium (DSN format changes)	Sentinel-aware DSNs everywhere (unverified)	~+3–6Gi memory + sentinel pods
E. Bitnami charts	—	subscription or unmaintained-image risk	—	—	not viable in 2026 without paying

5. Effort & recommended sequencing

Option	Effort
A — document & accept + mitigations	S (docs + values; PDB lands via #8 anyway)
B — operator, single node	M
C — operator, 3-node quorum	L (incl. app-side queue-type change + storage)
D — Redis Sentinel via operator	L (blocked on app-team Sentinel answer)
E — Bitnami	not recommended

Recommendation: A now; C later only if justified by an incident or an SLA.

1. Now (this issue's outcome): decision record in docs/ — Redis stays an ephemeral single-replica cache (documented rationale + failure-mode table from §3); RabbitMQ stays our single-node StatefulSet with durable classic queues. Follow-ups: priorityClass values per env, fast-storage recommendation, ensure [availability] PodDisruptionBudgets and topologySpreadConstraints #8 covers both pods.
2. Parallel, zero-commitment: ask the app team the two open questions — (a) does Shopsys's Redis client stack accept Sentinel DSNs end-to-end, (b) can Messenger transports move to quorum queues. Their answers gate C/D and cost nothing to collect.
3. Later, trigger-based: adopt the RabbitMQ Cluster Operator (B→C) only when a real availability incident, an SLA, or multi-node message-durability requirements appear. It is the one genuinely healthy adoption target here (official, free, actively maintained); the blockers are honest ones: cluster-wide prerequisite, credentials-secret remap, readiness-gate rework (wait: true on a CR release waits for the CR object, not cluster readiness), and NFS-vs-quorum storage.
4. Redis operators: revisit only if the cache stops being ephemeral. Spotahome is archived; OT-ContainerKit is alive but pre-1.0 — for a disposable cache, fast rescheduling beats Sentinel on every axis.

Sources: rabbitmq/cluster-operator releases · operator install docs · using the operator · RabbitMQ 4.0 release notes (CQ mirroring removed) · Quorum queues · Bitnami catalog changes · The New Stack on Bitnami · OT-CONTAINER-KIT/redis-operator · spotahome/redis-operator (archived) · Symfony RedisAdapter Sentinel · symfony#46600 (phpredis+Sentinel issue)