Skip to content

manager/orchestrator/restart: honor MaxAttempts for global jobs - #3299

Open
breken-ai wants to merge 1 commit into
moby:masterfrom
breken-ai:fix-global-job-restart-limit
Open

breken-ai wants to merge 1 commit into
moby:masterfrom
breken-ai:fix-global-job-restart-limit

Conversation

@breken-ai

Copy link
Copy Markdown

- What I did

Fixed --restart-max-attempts being ignored for global jobs. A global job whose task keeps failing is restarted forever. A replicated job with the same restart policy stops after the limit.

Supervisor.Restart creates the replacement task for a global job with the old task's node ID (orchestrator.NewTask(cluster, service, 0, t.NodeID)), so RecordRestartHistory stores the history under {Slot: 0, ServiceID, NodeID}. shouldRestart builds its lookup tuple with the node ID only when orchestrator.IsGlobalService(service) is true. That is false for GlobalJob mode, so the lookup uses {Slot: 0, ServiceID}, never finds the history, and shouldRestart always returns true.

- How I did it

shouldRestart now adds the node ID for global jobs too, the same as for global services. That matches the key Restart uses when it records the history. This is a one-line change in manager/orchestrator/restart/restart.go.

- How to test it

  • New spec in manager/orchestrator/jobs/orchestrator_restart_test.go, "should only restart global job tasks MaxAttempts times". It is the global-job version of the existing replicated-job MaxAttempts spec: one ready node, MaxAttempts: 3, and 4 failures, after which exactly 4 tasks should exist. On master it fails with 5 tasks because the 4th failure is restarted again. It passes with the fix.

  • go test ./manager/orchestrator/... passes. I also ran the jobs suite 15 times in separate processes and it passed every time. golangci-lint (v2.14.0) reports 0 issues on the changed packages, and gofmt is clean.

  • End to end: I built dockerd (linux/arm64) from docker-v29.8.1, once as is and once with this change copied into vendor/, and ran each in a docker:dind container:

    docker swarm init
    docker service create -d --name gj --mode global-job \
      --restart-condition on-failure --restart-max-attempts 2 --restart-delay 1s alpine:3.20 false
    # 40s later
    docker service ps -q gj | wc -l
    

    The unpatched build had 36 tasks and was still restarting. The patched build had 3 tasks (the original plus 2 restarts), all Failed. Stock Docker 29.8.1 behaves like the unpatched build, and a replicated-job with the same flags stops at 3 tasks on both builds.

- Description for the changelog

Fix global jobs ignoring the restart policy's max attempts and restarting failed tasks forever.


AI disclosure: an AI coding agent (Claude) found this bug, wrote the fix and test, and drafted this description. The breken-ai account submitted it, and every result above comes from commands that were actually run. The commit carries an Assisted-By: Claude trailer.

Restart records the restart history of a global-job task under a slot
tuple that includes the node ID, because the replacement task is created
for that node. shouldRestart only added the node ID for global services,
so for global jobs it looked the history up under a tuple without the
node ID, never found it, and always allowed another restart.

As a result, a global job with --restart-max-attempts restarted failing
tasks forever. Index global jobs by node ID in shouldRestart, like
global services.

Assisted-By: Claude
Signed-off-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant