Repository navigation
Conversation
The node error handler passed job label values to onPodError, which looks up a Pod by name. Jobs with distinct Pod names were skipped before the node was marked unschedulable. List every matching Pod with its actual namespace and name. Keep the existing per-pod recovery policy checks and job restart ledger so an eligible Pod can trigger recovery without duplicate job restarts. Signed-off-by: xinjun.jiang <xinjun.jiang@daocloud.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node error recovery passes Kubeflow job label values to
onPodError, which looks up a Pod by name. For a jobtrainingwith a Podtraining-worker-0, that lookup returns NotFound, and the job is never restarted before the node is marked unschedulable.Return each matching Pod's actual namespace and name. Evaluate every Pod's recovery policy so that a Pod with
RestartPolicyNeveror without recovery enabled does not hide an eligible sibling. The existing restart ledger keeps recovery to one restart per job and distinguishes jobs with the same name in different namespaces.The regression tests exercise the node event handler with a fake Kubernetes client, including node and label filtering, multiple Pods per job, namespace isolation, and mixed recovery policies. Keeping Pods available after the fake deletion also verifies ledger deduplication. Before the fix, all five cases that require recovery fail because no restart is requested; after the fix, all six cases pass.
Validation (Go 1.26.9, Helm 3.22.0):
make test-all(default and MetaX builds)go test -p 4 -race ./pkg/recovery -count=1go vet -p 4 ./...CGO_ENABLED=0 go build -p 4 ./cmd/kcovermake helm-testgofmtandgit diff --checkAll validation uses local fake clients and static Helm rendering. No cluster or GPU resources are required.