Skip to content

Keep runnable tasks queued across revisions - #3297

Open
Mazyod wants to merge 1 commit into
moby:masterfrom
Mazyod:53722-pending-task-requeue
Open

Mazyod wants to merge 1 commit into
moby:masterfrom
Mazyod:53722-pending-task-requeue

Conversation

@Mazyod

@Mazyod Mazyod commented Sep 18, 2026

Copy link
Copy Markdown

Fixes moby/moby#53722

Related to moby/moby#36699. Its later node-restart reports may share this cause, but lack the service-revision evidence needed to confirm that. The original issue concerns obsolete tasks desired Shutdown, whose behavior this patch preserves.

- What I did

Fix a valid Pending task being dropped from scheduler retries after a service-only metadata update. If no node can accept the task during a subsequent scheduler pass, the older-revision branch currently skips re-enqueueing it even though it is still desired Running. It can remain Pending after node recovery until a force update creates a replacement.

- How I did it

Require desired state Shutdown or later before taking the older-revision retirement branch in noSuitableNode. Still-runnable tasks reach the existing re-enqueue path. Add one regression with identical task specifications and different revision numbers, checking queue retention and assignment of the same task ID after Down becomes Ready.

- How to test it

Base: 1fd637ba5cc32ff30d1dd2bdb14997bd4f424b46; Go 1.27.1 linux/amd64.

  • Regression only on stock: go test ./manager/scheduler -run '^TestPendingOldRevisionRecoversWhenNodeReturns$' -count=1 fails because the task is absent from unassignedTasks.
  • With the fix: go test ./manager/scheduler -count=1 and go test -race ./manager/scheduler -count=1 pass. The existing TestUnscheduleableTask continues to verify retirement of an obsolete Pending task.
  • go test -parallel 8 -timeout 20m ./... -count=1 passes for the full root module, including integration, jobs, task reaper, and updater packages. A short temporary path is needed for the node tests' Unix sockets in a deeply nested checkout.
  • Fresh stock and patched Engine builds from Moby 29.8.1 were compared in disposable Linux amd64 DinD. Stock reproduces both pause/activate and worker stop/start; the patched manager recovers the same Pending task IDs in both, including with a stock worker. Only the identical vendored scheduler file was changed.

Runtime comparison: Moby 464cd50c3d9e92877d56940ea160de6fca7bea23, vendoring SwarmKit ad0357aedfca72c288abb8395ec5603e1136a9e5; Go 1.27.1, CGO_ENABLED=0, tags netgo osusergo static_build nri_no_wasm. Binary SHA-256: stock 34747d599445c61b0a79e1e7dc7bd175ce9f3a9885576cded4a6d16d7b46e6c6; fixed d2fd714b5dcddc237192cff9a1da44774e0874209da1b81593dc475cafecf495. The separate swarmd module and full Moby suite were not run.

- Description for the changelog

Keep valid pending tasks eligible for scheduling after service metadata updates and node recovery.

Created with: Astra (AI), under my supervision and review.

Require a desired shutdown state before retiring an older-revision task
when no node can accept it. Keep still-runnable Pending tasks queued,
and verify assignment of the same task after node recovery.

Fixes moby/moby#53722
Related to moby/moby#36699
Assisted-By: Astra (AI)

Signed-off-by: Mazyad Alabduljaleel <mazjaleel@gmail.com>

@koriyoshi2041 koriyoshi2041 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the scheduler ownership boundary at f54226a: only older-revision tasks already desired Shutdown or later take the retirement path; runnable and job-completion targets stay on the existing retry path. The regression also preserves the original task ID across node recovery, while TestUnscheduleableTask continues to cover obsolete Pending-task shutdown. Full scheduler tests and the race-enabled scheduler suite pass locally; diff hygiene is clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Swarm stops retrying a Pending task after a service-only label update

2 participants