Repository navigation
inventory: deploy cron wherever fluentd runs - #625
Merged
Merged
Conversation
The kolla cron container runs logrotate over /var/log/kolla. In inventory/51-kolla it is mapped to the common group (compute, control, monitoring, network, storage), while fluentd and kolla-logs are mapped to generic, i.e. every host. On a host that is in generic but in none of the common children, such as a dedicated loadbalancer node, fluentd receives the HAProxy syslog stream and writes /var/log/kolla/haproxy/haproxy_latest.log into the kolla_logs volume, but no cron container exists to rotate it. The file grows without limit until the root disk is full. Upstream kolla-ansible maps cron, fluentd and kolla_logs to the same group, so rotation always runs where logs are written. Map cron to generic as well to restore that pairing. As a result, cron is now also deployed to hosts that were only in generic, for example managers that are not in the monitoring group and dedicated loadbalancer nodes. UpgradeImpact Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
marked this pull request as ready for review
October 6, 2026 13:04
berendt
pushed a commit
to osism/container-image-kolla-ansible
that referenced
this pull request
Oct 7, 2026
Carry a patch that adds nocopytruncate to the logrotate block for /var/log/kolla/haproxy/*.log. On 2025.1 it changes the template in the common role, on 2025.2 the one in the cron role. 2026.1 replaced the per-service templates with a shared default, so there the patch adds a new cron-logrotate-haproxy.conf.j2, which the global template prefers over the default. kolla's global logrotate configuration uses copytruncate: the log is copied to .1 and the original truncated. The copy needs free space equal to the size of the log. When the log is larger than the free space, the copy fails with ENOSPC, logrotate removes the partial copy and leaves the original untouched. The log never shrinks, and every daily run fills the disk to 100% while it copies. That is exactly the state of hosts whose HAProxy log grew without rotation because they lacked the cron container: hosts in the generic inventory group but not in common, fixed by osism/generics#625. Once cron reaches them, copytruncate cannot rotate a log that already exceeds the free space. Renaming needs no free space. It is safe here because fluentd is the only writer: HAProxy sends its log via syslog, and fluentd's file output (append true) opens the file, appends a chunk and closes it again on every flush, then recreates the file on the next flush after a rename. The haproxy container does not mount kolla_logs. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kolla
croncontainer runs logrotate over/var/log/kolla. Ininventory/51-kollait is mapped tocommon(compute, control, monitoring, network, storage), whilefluentdandkolla-logsare mapped togeneric, i.e. every host.On a host that is in
genericbut in none of thecommonchildren, such as a dedicated loadbalancer node, fluentd receives HAProxy's syslog stream and writes/var/log/kolla/haproxy/haproxy_latest.loginto thekolla_logsvolume, but no cron container exists to rotate it. The file grows until the root disk is full.Upstream kolla-ansible maps
cron,fluentdandkolla_logsto the same group. This change mapscrontogeneric, restoring that pairing.Effect
cronis now also deployed to hosts that were only ingeneric, e.g. dedicated loadbalancer nodes and managers that are not inmonitoring. These hosts are already targeted by the samekolla-commonplaybook for thelogsandfluentdroles, so no new host category comes under kolla management; the addition is one crond container per host. Log directories of services not running on a host are skipped (missingok).Checked with
ansible-inventory --graphon an inventory with a dedicated loadbalancer node:fluentdcronUpgrade notes
Existing environments pick this up with
osism apply cron(orosism apply common). On a host where the log has already grown very large, delete it first: the first rotation usescopytruncate, which needs free space equal to the file size. fluentd recreates the file on its next flush.🤖 Generated with Claude Code