Deploy GHCR step refuses node drains when its pull-probe pod runs out of memory #5009

Activity

1. devantler

   added the

   Bug

   issue type on Oct 17, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Oct 17, 2026
3. coderabbitai commented on Oct 17, 2026

   

   coderabbitaiBot

   on Oct 17, 2026 – with coderabbitai

   Contributor

   More actions

   ⚠️ Possible Duplicate Issue(s)

   • fix(deploy): a worker with no free memory makes the credential probe fail and stops every production deploy #4664

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
4. devantler

   added a parent issue on Oct 17, 2026

   • Production deploys fail only for the change they deploy #4979
5. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Oct 17, 2026
6. devantler commented on Oct 24, 2026

   

   devantler

   on Oct 24, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer (interactive Claude Code session)

   Duplicate of #4664, which #4710 fixed. This issue's evidence window (2026-10-08 17:45Z to 2026-10-09 17:24Z) is the same incident #4664 recorded: run 37818995791 on 2026-10-08, a node-pinned probe refused with OutOfmemory on a worker with about 15 Mi left. #4664 was closed as completed on 2026-10-10 when #4710 merged; this issue was filed later from the failure count and names only the earlier, superseded attempt (#4666).

   Checked against this issue's criteria on platform main 1e8dad92 and, read-only, against production today:

   • Can the trigger still happen? The probe now asks for 1 MiB of memory and 1m CPU (limit 64 MiB), down from 16 MiB, with a comment citing fix(deploy): a worker with no free memory makes the credential probe fail and stops every production deploy #4664 (scripts/refresh-flux-ghcr-auth.sh); a fixture test pins those exact requests. Production's tightest node today has about 512 MiB of memory left to request, so the probe fits with a wide margin.
   • If it does happen, the step names the cause: a probe the kubelet refuses now fails at once with "did not run … not credential evidence" and the pod reason, instead of waiting out the budget and blaming the credential.
   • Self-test: TestRuntimeProbeKubeletRejectionFailsFastAndDistinctly drives an OutOfmemory rejection and asserts that message, and that the kubelet's raw message stays out of the log.
   • Heal path: it runs the same script, so it gets the same fix. No refusal of this kind has recurred since 2026-10-09.

   Reopen #4664, as its closing note asks, if a later deploy is refused at the probe on a full worker again.
7. devantler

   closed this as a duplicateof fix(deploy): a worker with no free memory makes the credential probe fail and stops every production deploy #4664 on Oct 24, 2026
8. github-project-automation

   moved this from 📥 Backlog to ✅ Done in 🌊 Project Boardon Oct 24, 2026