Skip to content

Keep node-local backup agents schedulable on saturated workers #4781

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Description

The node-local backup agent can remain unschedulable when a worker's memory requests are full. Its default scheduling priority cannot make room, although movable lower-priority workloads are present. A monitoring reservation refresh has already landed, so repeating that earlier repair does not resolve this newly observed agent admission failure.

Steps to Reproduce

  1. Render the pinned Velero chart with the production values.
  2. Observe that the node-agent has no declared priority tier.
  3. Restart an agent on a worker with insufficient unreserved memory and observe it remain pending.

Expected Behavior

The node-local backup agent must have schedulable capacity without lowering its memory request or displacing databases, storage clients, or critical infrastructure.

Actual Behavior

A current read-only observation shows an unschedulable backup agent. The server is available, but that does not establish complete node-local backup coverage.

Acceptance Criteria

  • Reproduce the missing scheduling tier with the actual pinned chart before changing configuration.
  • Reuse the existing modest storage priority tier for the production node-agent only; retain equal database priority and higher critical-infrastructure priority.
  • Preserve the single server, resource requests, backup inclusion, credentials, volumes, transfer settings, and bounded rolling update.
  • Pass regression controls, exact-head review and CI, then deliver through the normal protected merge queue after fix: unblock safe recovery of interrupted storage maintenance #4775.
  • Read back agent readiness and unchanged database redundancy; prove a subsequent full backup before declaring the broader backup incident recovered.

Part of #4723. This does not claim to fix the separately tracked source-pinned transfer admission behavior or close #4729's full-backup proof.

Incident/WIP exception: this is confirmed live backup-agent breakage, not proactive intake. The narrow repair is expedited while the already-reviewed maintenance recovery is validating in the queue. Private node identities, resource telemetry, and raw logs remain outside this issue.

Activity

  1. added theissue type on Oct 12, 2026
  2. moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Oct 12, 2026
  3. self-assigned this
    on Oct 12, 2026
  4. added a commit that references this issue on Oct 12, 2026
  5. moved this from 🏃🏻‍♂️ In Progress to 👀 In Review in 🌊 Project Boardon Oct 12, 2026
  6. moved this from 👀 In Review to ✅ Done in 🌊 Project Boardon Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions