Conversation
validateResources called genericresource.ValidateTask but returned nil when it failed, so a service reserving a named generic resource (for example NamedResourceSpec gpu=UUID1) was accepted. Tasks can only claim discrete generic resources, so the scheduler's ResourceFilter rejects every node for such a task and it stays pending forever with "insufficient resources", even on nodes that advertise that resource. Return the validation error as InvalidArgument. Assisted-By: Claude Signed-off-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
- What I did
CreateService/UpdateServicenow reject a service that reserves a named generic resource. Before this change such a service was accepted and could never be scheduled.validateResourcescallsgenericresource.ValidateTask, but when that check fails it returnsnilinstead of the error. This has been the case since generic resources were added in 85019da (2017). So a spec such aspasses validation. Tasks can only claim discrete generic resources:
genericresource.HasEnoughandClaimboth return "task should only hold Discrete type". The scheduler'sResourceFiltertherefore rejects every node, and the task staysPendingforever withno suitable node (insufficient resources on N nodes). That happens even when the node advertises exactly that resource. The docker CLI already refuses named generic resources forservice create/update, but API clients get no error, and the Engine API'sGenericResourcesexample showsNamedResourceSpecentries.- How I did it
validateResourcesnow returns theValidateTaskerror ascodes.InvalidArgument. This is a one-line change inmanager/controlapi/service.go.- How to test it
TestValidateResourcesandTestValidateResourceRequirementsgain a named generic resource case as a bad input, andTestValidateResourcesalso gains a discrete generic resource as a good input. The bad cases fail on master ("An error is expected but got nil") and pass with the fix.go test ./manager/controlapi/ ./manager/scheduler/ ./api/genericresource/passes. golangci-lint (v2.14.0) reports 0 issues onmanager/controlapiand gofmt is clean.dockerd(linux/arm64) fromdocker-v29.8.1, once as is and once with this change copied intovendor/. Each ran indocker:dindwith--node-generic-resource gpu=UUID1, and the service above was posted to/v1.51/services/create:HTTP 201, thennamed.1 Pending ... "no suitable node (insufficient resources on 1 node)", althoughdocker node inspect selflists{"NamedResourceSpec":{"Kind":"gpu","Value":"UUID1"}}HTTP 400 {"message":"rpc error: code = InvalidArgument desc = invalid argument for resource gpu"}- Description for the changelog
Reject services that reserve named generic resources instead of accepting them and leaving their tasks pending forever.
AI disclosure: an AI coding agent (Claude) found this bug, wrote the fix and test, and drafted this description. The
breken-aiaccount submitted it, and every result above comes from commands that were actually run. The commit carries anAssisted-By: Claudetrailer.