Stop unattended-upgrades from cancelling GitHub Actions jobs; patch warm-pool runners

[akuzminsky]

Two related fixes for the self-hosted actions-runner fleet, scoped to the runner role and synced across development, sandbox, and global copies (as #277 did). Design notes: .claude/plans/TASK-runner-unattended-upgrades-cancels-jobs.md.

1. unattended-upgrades no longer cancels running jobs

Symptom: GitHub Actions jobs on the runner fleet get cancelled mid-run (The operation was canceled) at random points. The runner instance stays healthy — it's the job that dies. Jobs killed mid-terraform apply leak AWS test resources.

Cause (from the runner's journal): Ubuntu's daily unattended-upgrades runs a needrestart pass that restarts every service linked against an updated library — including actions-runner.service. Restarting the runner mid-job cancels the job. Random kill point = apt-daily-upgrade.timer's large RandomizedDelaySec.

Fix (profile::github_runner::service):

• /etc/needrestart/conf.d/90-actions-runner.conf — a targeted $nrconf{override_rc} => 0 for actions-runner.service only. Patching continues, every other service still auto-restarts; only the runner is exempt.
• gha-restart-when-idle.timer — restarts the runner to load the patched libraries, but only when it's idle (no Runner.Worker) and a restart is actually pending (a (deleted) library mapping in the runner's process tree — ground truth, independent of how needrestart lists an exempted service). Re-checks idle right before acting.

2. Warm-pool runners are patched before they hibernate

The runner ASG uses a Hibernated warm pool. A hibernated instance never runs the daily upgrade timer, and a hibernation resume is not a boot (systemd boot units don't re-run), so pooled instances would resume and serve jobs on stale libraries.

Fix (profile::github_runner): apply pending security upgrades once during provisioning, before the instance hibernates, via a per-boot exec (tmpfs marker, written only on success so a failure retries). Fresh launches driven by max_instance_lifetime re-run it, bounding staleness. Doing it at provisioning — not at resume — keeps the fast hibernation resume untouched.

Does not conflict with Elasticsearch (verified)

profile::elastic::service also manages needrestart, so this was a deliberate concern:

• This lives in the runner role, not the shared profile::unattended_upgrades — never lands on ES nodes.
• Distinct conf.d filename (90-actions-runner.conf vs elastic's 90-elastic-no-auto-restart.conf).
• needrestart declared via stdlib::ensure_packages (not a bare package{}) — no duplicate-declaration collision.
• Elastic uses a global $nrconf{restart} = 'l'; the runner uses a targeted per-service override. Different mechanisms, and the two roles never share a host.

Testing

puppet-lint --fail-on-warnings clean on all four module paths; shellcheck clean on the idle-restart script; the needrestart drop-in validated as parseable Perl with the regex confirmed to match actions-runner.service.

Not yet exercised on a live runner — /proc/maps, pgrep, and hibernation mechanics are Linux-only. The real validation is: trigger an upgrade during a job on a runner and confirm the job survives, no Stopping actions-runner.service in the journal, and an idle runner with pending patches gets restarted.

One thing to weigh

The provisioning exec has timeout => 1200, which equals the ASG bootstrap lifecycle hook (heartbeat_timeout = 1200, default_result = ABANDON). The upgrade shares that provisioning budget; a weeks-stale AMI with a large security backlog could compete with the hook. Usually a non-issue on a fresh AMI (seconds–minutes), but worth capping the timeout lower if provisioning is tight.

🤖 Generated with Claude Code