add consumer autoscaling by RabbitMQ queue backlog declared in deploy/consumers.yaml

CLAUDE.md

@@ -50,10 +50,12 @@ DISPLAY_FINAL_CONFIGURATION=1 ./deploy/deploy-project.sh deploy
      - `environment-variables.sh`: Environment variable management
      - `deploy.sh`: Core deployment logic
      - `cron.sh`: Cron job configuration
-     - `autoscaling.sh`: Horizontal pod autoscaling
+     - `consumers.sh`: Consumer deployments and autoscalers from `deploy/consumers.yaml`
+     - `autoscaling.sh`: Horizontal pod autoscaling of php-fpm and storefront
 
 2. **Kubernetes Manifests** (`kubernetes/`)
    - `deployments/`: Application deployments (webserver, php-fpm, redis, rabbitmq, cron)
+   - `manifest-templates/`: Templates for generated manifests (consumer deployments and their autoscalers)
    - `configmap/`: Configuration files for services
    - `services/`: Kubernetes service definitions
    - `kustomize/`: Kustomization overlays for different deployment scenarios

README.md

@@ -111,6 +111,7 @@ If you want to define your custom variables see [Define custom variables](#defin
 | USING_CLOUDFLARE             | _1_ OR _0_                       | Set to 1 if your site is using Cloudflare (enables IP whitelisting)                                                         | production/devel |
 | MCP_INGRESS_ENABLED          | _1_ OR _0_                       | Set to 0 to disable the separate ingress publishing the MCP endpoints without HTTP basic auth (default: 1)                  | production/devel |
 | MCP_IP_WHITELIST             | 203.0.113.0/24, 198.51.100.10/32 | VPN egress IP ranges allowed to access MCP; when empty, MCP access is not restricted by source IP                           | production/devel |
+| ENABLE_CONSUMER_AUTOSCALING  | _true_ OR _false_                | Enable autoscaling of consumers by RabbitMQ queue backlog (default: false), see [Consumers](#consumers)                     | production/devel |
 
 *1) Credentials can be generated in Gitlab (Settings -> Repository -> Deploy Tokens) with `read_registry` scope only 
 
@@ -146,6 +147,98 @@ You can override Kubernetes manifests by placing your custom manifests into `app
        ...
    ```
 
+### Consumers
+
+Consumers are Symfony Messenger workers (`messenger:consume`) deployed as `consumer-<name>` deployments.
+By default they are declared in the `DEFAULT_CONSUMERS` array in `deploy-project.sh` in the format `<name>:<transports separated by space>:<replicas>`,
+e.g. `"product-recalculation:product_recalculation_priority_high product_recalculation_priority_regular:1"`.
+This way keeps working unchanged, but it cannot declare autoscaling - for that declare the consumers in `consumers.yaml` instead.
+
+#### Declare consumers in consumers.yaml
+
+1. Source the new part in the `deploy()` function of `deploy-project.sh` before `environment-variables.sh`,
+   which injects the environment variables into the generated consumer deployments (the part fails the deploy when sourced too late):
+
+   ```diff
+   ...
+       source "${DEPLOY_TARGET_PATH}/parts/domain-rabbitmq-management.sh"
+   +   source "${DEPLOY_TARGET_PATH}/parts/consumers.sh"
+       source "${DEPLOY_TARGET_PATH}/parts/environment-variables.sh"
+   ...
+   ```
+
+2. Move the consumer declaration from `DEFAULT_CONSUMERS` to `app/deploy/consumers.yaml` and remove the array from `deploy-project.sh`:
+
+   ```yaml
+   consumers:
+       -   name: product-recalculation                   # deployment is named consumer-<name>
+           transports: [product_recalculation_priority_high, product_recalculation_priority_regular]
+           replicas: 1                                   # static replicas count, used when consumer autoscaling is disabled
+           autoscaling:                                  # optional, omit for a consumer with static replicas only
+               minReplicas: 1                            # 0 allowed on a cluster with the HPAScaleToZero feature gate
+               maxReplicas: 8                            # must be higher than minReplicas
+               threshold: 500                            # target count of ready messages per pod
+               queues: [product_recalculation_priority_high, product_recalculation_priority_regular]   # optional, defaults to transports
+       -   name: email
+           transports: [email_transport]
+           replicas: 1
+   ```
+
+   | Field                     | Meaning                                                                                                                                                       |
+   |:--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+   | `name`                    | Name of the deployment `consumer-<name>`: lowercase letters, digits and dashes, starting and ending with a letter or digit, at most 54 characters               |
+   | `transports`              | Non-empty list of Symfony Messenger transport names passed to `messenger:consume` (same character rules as `autoscaling.queues`)                              |
+   | `replicas`                | Static replicas count (integer, `0` allowed) used when consumer autoscaling is disabled                                                                       |
+   | `autoscaling`             | Optional, omit it for a consumer that should always run with the static `replicas`                                                                            |
+   | `autoscaling.minReplicas` | Lower bound of the autoscaler; `0` scales the consumer to zero pods while its queues are empty and needs the `HPAScaleToZero` feature gate on the cluster (the API server rejects `0` without it) |
+   | `autoscaling.maxReplicas` | Upper bound of the autoscaler, must be higher than `minReplicas`                                                                                              |
+   | `autoscaling.threshold`   | Target count of ready messages per pod, see [Enable consumer autoscaling](#enable-consumer-autoscaling)                                                        |
+   | `autoscaling.queues`      | Non-empty list of RabbitMQ queue names watched by the autoscaler, defaults to `transports` - set it when the queue name differs from the transport name (see `config/packages/messenger.yaml`); letters, digits, `_`, `.` and `-`, starting and ending with a letter or digit, at most 63 characters |
+
+> [!IMPORTANT]
+> A project must use either `DEFAULT_CONSUMERS` or `consumers.yaml`. The deploy fails when `consumers.yaml` exists
+> and the merge phase already generated consumer deployments (from `DEFAULT_CONSUMERS` or from `orchestration/kubernetes/deployments`).
+
+The file is read during deploy, so changing replicas or thresholds does not need a rebuild of the image.
+The whole declaration, including the autoscaling blocks, is validated on every deploy even when
+`ENABLE_CONSUMER_AUTOSCALING` is not set. A mistake in the file therefore fails on any environment,
+not only on the one with autoscaling enabled.
+
+#### Enable consumer autoscaling
+
+Consumers with an `autoscaling` block can be scaled by the count of messages waiting in their RabbitMQ queues instead of running a static number of replicas.
+Each of them gets a Horizontal pod autoscaler (`autoscaling/v2`) over the external metric `rabbitmq_queue_backlog` (ready messages only),
+summed over all `queues` of the consumer.
+
+> [!IMPORTANT]
+> The autoscalers need the external metric `rabbitmq_queue_backlog` (label `queue`, namespace-scoped)
+> from the External Metrics API of the cluster. The Shopsys clusters provide it. Without the metric
+> the autoscaler reports `ScalingActive: False`, keeps its consumer at `minReplicas` and never scales it up.
+
+Set `ENABLE_CONSUMER_AUTOSCALING=true` as an environment variable of the environments that should scale (e.g. production only, default is `false`):
+
+- enabled: an autoscaler is deployed for every consumer with an `autoscaling` block and `replicas` is omitted from its deployment, so the autoscaler owns the replicas count
+- disabled: consumers run with the static `replicas`, so the variable works as a kill switch
+
+Existing autoscalers are updated in place by the deploy. An autoscaler of a consumer that lost its `autoscaling` block, was renamed
+or whose environment disabled the flag is deleted after the successful build of the migrate-application configuration and right before its apply
+(kubectl apply does not prune), a deploy failing earlier never touches them. The cleanup needs `list` and `delete` permissions
+on `horizontalpodautoscalers` in the namespace for the deploy account and runs only for projects with `consumers.yaml`.
+A renamed consumer leaves its old deployment `consumer-<old name>` behind for the same reason, delete it manually (this applies to `DEFAULT_CONSUMERS` as well),
+and so do the autoscalers of a project that removes `consumers.yaml` altogether.
+
+> [!NOTE]
+> The first deploy of a consumer with an autoscaler (after enabling the flag or after adding its `autoscaling` block) resets it to 1 replica for a moment: removing `replicas` from a deployment
+> that was previously applied with a static count makes the API server fall back to the default, until the new autoscaler reconciles (within its 15 s sync period).
+> Do it outside of peak hours if that matters. Later deploys keep the replicas set by the autoscaler.
+
+`threshold` is the target count of ready messages per pod. A useful rule of thumb is the count of messages one pod processes in about a minute (per-pod throughput × 60 s):
+with a threshold of `500` and 2000 ready messages the autoscaler runs 4 pods, with an empty queue it scales down to `minReplicas`.
+Scale-up is immediate, scale-down starts after 5 minutes of stabilization and removes at most 1 pod per 2 minutes.
+
+The scaling behavior and the metric name live in `kubernetes/manifest-templates/consumer-hpa.template.yaml` and can be overridden in `orchestration/kubernetes/manifest-templates/` as any other manifest.
+Keep the label `consumer-autoscaling: "true"` in an overridden template, the deploy recognizes the autoscalers it manages (and deletes the stale ones) by it.
+
 ### Add more or less domains
 
 > This example will work with 3 domains

UPGRADE.md

@@ -10,6 +10,21 @@
 2. Run `composer update shopsys/deployment`
 3. Check files in mentioned pull requests and if you have any of them extended in your project, apply changes manually
 
+## Upgrade from v5.4.0 to v5.5.0
+
+- consumers can be declared in `deploy/consumers.yaml` and autoscaled by RabbitMQ queue backlog, see [Consumers](README.md#consumers)
+  - to use it, source the new part in the `deploy()` function of `deploy-project.sh` (before `environment-variables.sh`, the part fails the deploy otherwise) and move the consumer declaration from `DEFAULT_CONSUMERS` to the file:
+    ```diff
+        source "${DEPLOY_TARGET_PATH}/parts/domain-rabbitmq-management.sh"
+    +   source "${DEPLOY_TARGET_PATH}/parts/consumers.sh"
+        source "${DEPLOY_TARGET_PATH}/parts/environment-variables.sh"
+    ```
+  - `DEFAULT_CONSUMERS` keeps working unchanged, nothing changes for projects that do not adopt the file; a project using both fails the deploy
+  - the deploy account of a project adopting the file needs `list` and `delete` permissions on `horizontalpodautoscalers` in the project namespace (autoscalers of consumers no longer autoscaled are deleted by the deploy)
+  - `autoscaling.minReplicas: 0` scales a consumer to zero pods while its queues are empty; it needs the `HPAScaleToZero` feature gate on the cluster, ask your cluster administrator
+  - the autoscaling is disabled by default, enable it per environment with `ENABLE_CONSUMER_AUTOSCALING=true`; the first deploy after enabling it briefly resets the autoscaled consumers to 1 replica, see [Enable consumer autoscaling](README.md#enable-consumer-autoscaling)
+  - if you override `deploy/parts/deploy.sh` in your project, apply the change from the pull request manually (the migrate-application kustomization is built into a file first, then autoscalers of consumers that are no longer autoscaled are deleted and the built file is applied)
+
 ## Upgrade from v5.3.0 to v5.4.0
 
 - added Gotenberg service for gift voucher PDF rendering ([#83](https://github.com/shopsys/deployment/pull/83))

deploy/functions.sh

@@ -85,23 +85,92 @@ function remove_dist() {
 
 function create_consumer_manifests() {
     local -a DEFAULT_CONSUMERS=("$@")
-
-    TEMPLATE_PATH="${CONFIGURATION_TARGET_PATH}/manifest-templates/consumer.template.yaml"
+    local CONSUMER NAME TRANSPORT_NAMES REPLICAS_COUNT
 
     for CONSUMER in "${DEFAULT_CONSUMERS[@]}"; do
         IFS=":" read -r NAME TRANSPORT_NAMES REPLICAS_COUNT <<< "$CONSUMER"
 
-        CONSUMER_MANIFEST_PATH="${CONFIGURATION_TARGET_PATH}/deployments/consumer-${NAME}.yaml"
+        create_consumer_deployment_manifest "${NAME}" "${TRANSPORT_NAMES}" "${REPLICAS_COUNT}"
+    done
+}
+
+function create_consumer_deployment_manifest() {
+    local NAME="$1"
+    local TRANSPORT_NAMES="$2"
+    local REPLICAS_COUNT="$3"
+
+    local TEMPLATE_PATH="${CONFIGURATION_TARGET_PATH}/manifest-templates/consumer.template.yaml"
+    local CONSUMER_MANIFEST_PATH="${CONFIGURATION_TARGET_PATH}/deployments/consumer-${NAME}.yaml"
+
+    cp "${TEMPLATE_PATH}" "${CONSUMER_MANIFEST_PATH}"
+
+    sed -i "s|{{NAME}}|${NAME}|g" "${CONSUMER_MANIFEST_PATH}"
+    sed -i "s|{{TRANSPORT_NAMES}}|${TRANSPORT_NAMES}|g" "${CONSUMER_MANIFEST_PATH}"
+    sed -i "s|{{REPLICAS_COUNT}}|${REPLICAS_COUNT}|g" "${CONSUMER_MANIFEST_PATH}"
+
+    add_migrate_application_resource "../../../deployments/consumer-${NAME}.yaml"
+}
+
+# Adds a manifest (path relative to the kustomization directories) to all migrate-application kustomizations
+function add_migrate_application_resource() {
+    local RESOURCE_PATH="$1"
+    local DEPLOY_TYPE
+
+    for DEPLOY_TYPE in continuous-deploy first-deploy first-deploy-with-demo-data; do
+        sed -i "/resources:/a\    - ${RESOURCE_PATH}" "${CONFIGURATION_TARGET_PATH}/kustomize/migrate-application/${DEPLOY_TYPE}/kustomization.yaml"
+    done
+}
+
+function create_consumer_hpa_manifest() {
+    local NAME="$1"
+    local MIN_REPLICAS="$2"
+    local MAX_REPLICAS="$3"
+    local SCALE_THRESHOLD="$4"
+    local QUEUE_NAMES_LIST="$5"
+
+    local TEMPLATE_PATH="${CONFIGURATION_TARGET_PATH}/manifest-templates/consumer-hpa.template.yaml"
+    local CONSUMER_HPA_MANIFEST_PATH="${CONFIGURATION_TARGET_PATH}/autoscaling/consumer-${NAME}.yaml"
 
-        cp "${TEMPLATE_PATH}" "${CONSUMER_MANIFEST_PATH}"
+    mkdir -p "${CONFIGURATION_TARGET_PATH}/autoscaling"
+    cp "${TEMPLATE_PATH}" "${CONSUMER_HPA_MANIFEST_PATH}"
 
-        sed -i "s|{{NAME}}|${NAME}|g" "${CONSUMER_MANIFEST_PATH}"
-        sed -i "s|{{TRANSPORT_NAMES}}|${TRANSPORT_NAMES}|g" "${CONSUMER_MANIFEST_PATH}"
-        sed -i "s|{{REPLICAS_COUNT}}|${REPLICAS_COUNT}|g" "${CONSUMER_MANIFEST_PATH}"
+    local QUEUE_NAMES="" QUEUE_NAME
+    for QUEUE_NAME in ${QUEUE_NAMES_LIST}; do
+        if [ -n "${QUEUE_NAMES}" ]; then
+            QUEUE_NAMES="${QUEUE_NAMES}, "
+        fi
+        QUEUE_NAMES="${QUEUE_NAMES}\"${QUEUE_NAME}\""
+    done
+
+    sed -i "s|{{NAME}}|${NAME}|g" "${CONSUMER_HPA_MANIFEST_PATH}"
+    sed -i "s|{{MIN_REPLICAS}}|${MIN_REPLICAS}|g" "${CONSUMER_HPA_MANIFEST_PATH}"
+    sed -i "s|{{MAX_REPLICAS}}|${MAX_REPLICAS}|g" "${CONSUMER_HPA_MANIFEST_PATH}"
+    sed -i "s|{{SCALE_THRESHOLD}}|${SCALE_THRESHOLD}|g" "${CONSUMER_HPA_MANIFEST_PATH}"
+    sed -i "s|{{QUEUE_NAMES}}|${QUEUE_NAMES}|g" "${CONSUMER_HPA_MANIFEST_PATH}"
+
+    add_migrate_application_resource "../../../autoscaling/consumer-${NAME}.yaml"
+}
 
-        sed -i "/resources:/a\    - ../../../deployments/consumer-${NAME}.yaml" "${CONFIGURATION_TARGET_PATH}/kustomize/migrate-application/continuous-deploy/kustomization.yaml"
-        sed -i "/resources:/a\    - ../../../deployments/consumer-${NAME}.yaml" "${CONFIGURATION_TARGET_PATH}/kustomize/migrate-application/first-deploy/kustomization.yaml"
-        sed -i "/resources:/a\    - ../../../deployments/consumer-${NAME}.yaml" "${CONFIGURATION_TARGET_PATH}/kustomize/migrate-application/first-deploy-with-demo-data/kustomization.yaml"
+# Builds a kustomization into a file: runCommand appends 2>&1 to its command, so a redirect written inline would send the kustomize error into the file
+function build_kustomization() {
+    local KUSTOMIZE_PATH="$1"
+    local OUTPUT_PATH="$2"
+
+    kustomize build --load-restrictor LoadRestrictionsNone "${KUSTOMIZE_PATH}" > "${OUTPUT_PATH}"
+}
+
+# Deletes the consumer autoscalers (label consumer-autoscaling=true) deployed in the namespace that are not in the built manifest (first argument).
+function delete_stale_consumer_hpas() {
+    local MANIFEST_PATH="$1"
+    local WANTED_HPAS DEPLOYED_HPAS DEPLOYED_HPA
+
+    WANTED_HPAS=$(yq e -N 'select(.kind == "HorizontalPodAutoscaler" and .metadata.labels["consumer-autoscaling"] == "true") | .metadata.name' "${MANIFEST_PATH}") || return 1
+    DEPLOYED_HPAS=$(kubectl get hpa -l consumer-autoscaling=true --namespace="${PROJECT_NAME}" -o name) || return 1
+
+    for DEPLOYED_HPA in ${DEPLOYED_HPAS}; do
+        if ! grep -Fqx -- "${DEPLOYED_HPA##*/}" <<< "${WANTED_HPAS}"; then
+            kubectl delete "${DEPLOYED_HPA}" --namespace="${PROJECT_NAME}" || return 1
+        fi
     done
 }