Delete live Kubernetes task Jobs after pre-start failure - #134
Draft
warp-agent-staging[bot] wants to merge 2 commits into
Draft
Delete live Kubernetes task Jobs after pre-start failure#134warp-agent-staging[bot] wants to merge 2 commits into
warp-agent-staging[bot] wants to merge 2 commits into
Conversation
Unschedulable timeout left the Job Pending, so TTL never started and autoscaler could still boot the agent against an already-ended execution. Delete Jobs that fail before JobFailed, and make CancelTask delete the Job. Preserve NoCleanup, finished-JobFailed post-mortem retention, and ExecuteTask context-cancel retention for worker disruption.
Contributor
Author
|
This PR was generated with Warp. Comment    |
Keep post-mortem TTL cleanup when the task container already exited before JobFailed arrives. Bound ExecuteTask Job deletion with an independent BackendShutdownTimeout so a wedged API server cannot block the worker forever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes REMOTE-3079: after an unschedulable timeout the Kubernetes backend reported
ExecuteTaskfailure but left the still-Pending Job in place.TTLSecondsAfterFinishednever starts until the Job completes, so cluster autoscaler could later schedule the orphaned pod against an already-ended execution.Changes
NoCleanupis set.JobFailedJobs, and Jobs whosetaskcontainer already terminated, for post-mortem TTL cleanup. Successful Jobs still delete immediately.context.WithTimeout(context.Background(), BackendShutdownTimeout)so a wedged API server cannot block the worker.KubernetesBackend.CancelTaskto delete the Job (no-op whenNoCleanupis set).ExecuteTaskcontext cancellation still only stops local watching so worker disruption can preserve running Jobs.Verification
gofmt -sclean on the changed filesgo vet ./...go test ./...go build -v ./...taskcontainer beforeJobFailedretains the Job; blocking delete returns atBackendShutdownTimeout;NoCleanupretains unschedulable Jobs;CancelTaskdeletes (and is a no-op for missing Jobs /NoCleanup); existing success,JobFailed, and context-cancel retention tests still passREMOTE-2831 (retry/timeout policy for pre-start k8s infra failures) is complementary and out of scope.