Conversation
Restart records the restart history of a global-job task under a slot tuple that includes the node ID, because the replacement task is created for that node. shouldRestart only added the node ID for global services, so for global jobs it looked the history up under a tuple without the node ID, never found it, and always allowed another restart. As a result, a global job with --restart-max-attempts restarted failing tasks forever. Index global jobs by node ID in shouldRestart, like global services. Assisted-By: Claude Signed-off-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
- What I did
Fixed
--restart-max-attemptsbeing ignored for global jobs. A global job whose task keeps failing is restarted forever. A replicated job with the same restart policy stops after the limit.Supervisor.Restartcreates the replacement task for a global job with the old task's node ID (orchestrator.NewTask(cluster, service, 0, t.NodeID)), soRecordRestartHistorystores the history under{Slot: 0, ServiceID, NodeID}.shouldRestartbuilds its lookup tuple with the node ID only whenorchestrator.IsGlobalService(service)is true. That is false forGlobalJobmode, so the lookup uses{Slot: 0, ServiceID}, never finds the history, andshouldRestartalways returns true.- How I did it
shouldRestartnow adds the node ID for global jobs too, the same as for global services. That matches the keyRestartuses when it records the history. This is a one-line change inmanager/orchestrator/restart/restart.go.- How to test it
New spec in
manager/orchestrator/jobs/orchestrator_restart_test.go, "should only restart global job tasks MaxAttempts times". It is the global-job version of the existing replicated-job MaxAttempts spec: one ready node,MaxAttempts: 3, and 4 failures, after which exactly 4 tasks should exist. On master it fails with 5 tasks because the 4th failure is restarted again. It passes with the fix.go test ./manager/orchestrator/...passes. I also ran the jobs suite 15 times in separate processes and it passed every time. golangci-lint (v2.14.0) reports 0 issues on the changed packages, and gofmt is clean.End to end: I built
dockerd(linux/arm64) fromdocker-v29.8.1, once as is and once with this change copied intovendor/, and ran each in adocker:dindcontainer:The unpatched build had 36 tasks and was still restarting. The patched build had 3 tasks (the original plus 2 restarts), all
Failed. Stock Docker 29.8.1 behaves like the unpatched build, and areplicated-jobwith the same flags stops at 3 tasks on both builds.- Description for the changelog
Fix global jobs ignoring the restart policy's max attempts and restarting failed tasks forever.
AI disclosure: an AI coding agent (Claude) found this bug, wrote the fix and test, and drafted this description. The
breken-aiaccount submitted it, and every result above comes from commands that were actually run. The commit carries anAssisted-By: Claudetrailer.