Skip to content

CI/CD: Add container memory, OOM kill and restart alarms and reduce noisy alarms - #3588

Merged
denniskuijs merged 8 commits into
mainfrom
bugfix/cloudformation
Nov 20, 2026
Merged

denniskuijs merged 8 commits into
mainfrom
bugfix/cloudformation

Conversation

@richturner

@richturner richturner commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Description

Adds Docker container memory monitoring to cloudformation-create-ec2.yml, so a container approaching its memory limit, or being killed for exceeding it, raises an alarm. It also tunes several existing alarms so they fire on sustained problems rather than brief, normal spikes.

A docker-metrics systemd timer runs /usr/local/bin/push-docker-metrics every minute (only when metrics are enabled) and publishes to the Custom/Docker namespace:

Metric Dimensions Meaning
ContainerMemoryPercent InstanceName, Container Anonymous memory as a percentage of the container's memory limit
MaxContainerMemoryPercent InstanceName Highest ContainerMemoryPercent on the instance
ContainerOomKills InstanceName Processes killed by the OOM killer inside any container since the last run
ContainerRestarts InstanceName Container restarts since the last run

Alarms:

  • ContainerMemoryUtilizationAlarm: MaxContainerMemoryPercent above 85% for 5 minutes. ContainerMemoryPercent shows which container.
  • ContainerOomKillAlarm: any OOM kill inside a container, e.g. a PostgreSQL backend.
  • ContainerRestartAlarm: any container restart, e.g. a crash, or a JVM whose main process was OOM killed.

Why not docker stats

docker stats MemPerc includes active page cache and shared memory. PostgreSQL relies on both (shared_buffers plus the OS cache of its data files, which effective_cache_size assumes), so a healthy PostgreSQL container sits close to its limit and kept a docker stats based alarm above 85%. That memory is reclaimable and doesn't lead to an OOM kill. The script reads anon from each container's cgroup v2 memory.stat instead. For the JVM containers memory is almost entirely anonymous, so their values are unchanged.

OOM kills and restarts are reported as counts since the previous run, using counters stored in /run/docker-metrics/counters. An OOM kill of a container's main process restarts the container, which recreates its cgroup (resetting oom_kill) and clears Docker's OOMKilled flag. That case is only visible as a restart, hence the separate restart metric. Containers without a memory limit are skipped for the memory percentages.

Alarm tuning

Several existing alarms fired on a single 5-minute datapoint for conditions that are normal when brief. They now need the condition to last:

Alarm Before After Why
DeadTupleAlarm 1 × 5 min, > 20% 12 × 5 min, > 30% Dead tuples build up between autovacuum runs, and autovacuum only starts at 20% dead by default, so the old alarm fired before every vacuum. Only a table staying high for an hour means vacuum is blocked or not keeping up.
LongRunningQueriesAlarm 1 × 5 min 3 × 5 min The nightly datapoint export and TimescaleDB compression jobs could trip it with a single slow query.
PostgreSQLConnectionsAlarm 1 × 5 min 3 × 5 min Ignores short connection bursts.
AttributeEventAlarm 1 period, band 2 2 of 3 periods, band 3 A band two standard deviations wide tripped on ordinary traffic changes. Missing data still breaches, so a stopped manager is still reported, within 10 minutes.

TableBloatCountAlarm removed. pg_table_bloat_count counts tables over 5 MB whose dead tuples exceed 30% of live tuples. That is the same condition DeadTupleAlarm checks per table from pg_dead_tuple_percent, so the two fired together. The bloat count also flapped as dead tuples crossed 30% between vacuum runs. DeadTupleAlarm is the better alarm of the two because it names the table. The metric is still collected for the dashboard.

DeadTupleAlarm is designed to work with #3599, which makes pg_dead_tuple_percent report 0 for tables with fewer than 10,000 dead tuples. Without it, small tables such as dashboard can still sit above 30% for an hour.

Other changes

  • daily-cleanup cron file: removed the #!/bin/bash line; cron files are not scripts, so it was only a stray comment
  • deployment_local_dir: mkdir -p, so re-running cfn-init doesn't fail when the directory exists
  • cfn-hup hook renamed to /etc/cfn/hooks.d/cfn-auto-reloader.conf, as it reloads all of AWS::CloudFormation::Init, not only the CloudWatch agent

Testing

The template parses (alarm settings checked after the tuning changes), and the rendered script was run against real docker ps/docker inspect output with a cgroup v2 tree laid out as on AL2023. It produces valid put-metric-data JSON with the expected percentages, skips containers without a limit, and reports OOM kill and restart deltas correctly, including a container that was really OOM killed and restarted, and an oom_kill counter reset. It hasn't yet been run on an EC2 instance.

Checklist

  • 1. Acceptance criteria of the linked issue(s) are met
  • 2. Tests are written and all tests pass
  • 3. Changes are manually tested by you and the reviewer
  • 4. Documentation is written or updated

@richturner

Copy link
Copy Markdown
Member Author

After adding the docker metric service to the EC2 template it was discovered that cfn-hup was not working on our current instances so this PR also includes a fix for that. Once the Cloudformation template is updated the following command should be run on the instance to fix the broken cfn-hup service, the docker metric service will then be loaded by cfn-hup when it next runs (~10mins):

sudo unlink /etc/systemd/system/cfn-hup.service
sudo tee /etc/systemd/system/cfn-hup.service > /dev/null <<'EOF'
[Unit]
Description=CloudFormation helper daemon
After=network.target

[Service]
Type=simple
ExecStart=/opt/aws/bin/cfn-hup --no-daemon --verbose
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo chmod 0644 /etc/systemd/system/cfn-hup.service
sudo systemctl daemon-reload
sudo systemctl enable --now cfn-hup

@richturner richturner self-assigned this Nov 18, 2026
@richturner
richturner requested a review from a team November 18, 2026 14:56
@denniskuijs
denniskuijs self-requested a review November 18, 2026 15:12
@denniskuijs denniskuijs self-assigned this Nov 18, 2026
@denniskuijs
denniskuijs removed the request for review from a team November 18, 2026 21:23
@denniskuijs

denniskuijs commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

cfn-hup isn't working because the symlink is still pointing to the old version after the cfn-bootstrap package was updated to a newer version.

Version installed:

rpm -q aws-cfn-bootstrap
aws-cfn-bootstrap-2.0-39.amzn2023.noarch

Broken symlink

ls -l /etc/systemd/system/cfn-hup.service
lrwxrwxrwx. 1 root root 62 Oct 30  2025 /etc/systemd/system/cfn-hup.service -> /opt/aws/apitools/cfn-init-2.0-36/init/systemd/cfn-hup.service

Steps to fix:

sudo rm /etc/systemd/system/cfn-hup.service

sudo systemctl daemon-reload

sudo systemctl enable cfn-hup
sudo systemctl start cfn-hup
sudo systemctl status cfn-hup

@richturner

richturner commented Nov 20, 2026 •

Copy link
Copy Markdown
Member Author

@denniskuijs good work finding the root cause, let's go with the remediation steps you describe as there's no need to manually write the service file as in my comment. These steps need to be manually performed on existing instances which would need coordinating.

The adjustments in this PR are still relevant I believe and might need to rename the PR

@denniskuijs

denniskuijs commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

@richturner I've reverted the cfn-hup changes (b7e02cb) in this PR as they're no longer needed. The other changes are still relevant, so I'll remove 'Fix broken EC2 cfn-hup service' from the PR title.

Please let me know if this PR is now ready for approval.

@denniskuijs denniskuijs changed the title CI/CD: Fix broken EC2 cfn-hup service and add docker mem usage metric CI/CD: Add Docker mem usage metric Nov 20, 2026
@denniskuijs denniskuijs changed the title CI/CD: Add Docker mem usage metric CI/CD: Add Docker memory usage metric Nov 20, 2026
…d restarts

docker stats MemPerc includes active page cache and shared memory, so the
PostgreSQL container, whose shared_buffers and cached data files fill its
limit by design, kept the alarm above 85% while healthy. Read anonymous memory
from each container's cgroup instead and publish it per container
(ContainerMemoryPercent) as well as the instance maximum
(MaxContainerMemoryPercent) used by the existing alarm, which now needs 5
minutes above the threshold.

Add ContainerOomKills and ContainerRestarts, published as the number seen
since the previous run, with alarms on any occurrence. An OOM kill of a
container's main process (e.g. a JVM) restarts the container and resets its
cgroup counters and OOMKilled flag, so it is only visible as a restart.
@richturner richturner changed the title CI/CD: Add Docker memory usage metric CI/CD: Add Docker container memory, OOM kill and restart metrics and alarms Nov 20, 2026
@richturner

Copy link
Copy Markdown
Member Author

@denniskuijs Added more alarms and some refactoring of existing ones and a description for this PR.

I have left the cfn-hup conf rename in place as it aligns with the AWS doc naming convention and makes more sense

…tAlarm

Several alarms fired on a single 5-minute datapoint for conditions that are
normal when brief:

- DeadTupleAlarm: dead tuples build up between autovacuum runs, so require 12
  periods (1 hour) above 30% instead of one period above 20%, which matched
  the default autovacuum trigger and fired before every vacuum
- LongRunningQueriesAlarm: 3 periods, so the nightly export and compression
  jobs don't trip it with a single slow query
- PostgreSQLConnectionsAlarm: 3 periods, to ignore short connection bursts
- AttributeEventAlarm: band width 3 and 2 of 3 periods, so ordinary traffic
  changes don't trip it; missing data still breaches, so a stopped manager
  is still reported

TableBloatCountAlarm is removed: pg_table_bloat_count counts tables over 5 MB
with dead tuples above 30% of live tuples, which is the same signal
DeadTupleAlarm raises per table. The metric is still collected for the
dashboard.
@richturner richturner changed the title CI/CD: Add Docker container memory, OOM kill and restart metrics and alarms CI/CD: Add container memory, OOM kill and restart alarms and reduce noisy alarms Nov 20, 2026
@richturner

Copy link
Copy Markdown
Member Author

Verified this updated template on ****sten prod

@denniskuijs

denniskuijs commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

@richturner Looks good! I've indeed renamed the cfn-hup configuration file to align with the AWS documentation

denniskuijs
denniskuijs previously approved these changes Nov 20, 2026
@sonarqubecloud

Copy link
Copy Markdown

@denniskuijs
denniskuijs added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 973c86e Nov 20, 2026
21 checks passed
@denniskuijs
denniskuijs deleted the bugfix/cloudformation branch November 20, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants