Repository navigation
fix(aws-batch): multi-node job nodes and Fargate platform checks #1684
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,8 @@ | ||
| - Jobs run no workload: a job always succeeds (container `exitCode` 0) once its dependencies succeed. By default it is `SUCCEEDED` as soon as it is submitted; with `--async-settle` it spends about one second each in `SUBMITTED`, `RUNNABLE` and `STARTING`, then five seconds in `RUNNING` (and stays `PENDING` while a dependency is unfinished). `attempts` is always empty, and retry strategies and timeouts are stored and echoed but never applied. | ||
| - `CancelJob` on a `PENDING` job takes effect at once. Real Batch waits for the job's dependencies to finish first. | ||
| - Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. Multi-node parallel jobs run as a single job: `ListJobs` with `multiNodeJobId` returns an empty list. | ||
| - Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. | ||
| - Multi-node parallel jobs: nodes are derived from the job, not stored one by one. `DescribeJobs` on `<jobId>#<n>` and `ListJobs` with `multiNodeJobId` return each node with `nodeDetails` (`nodeIndex`, `isMainNode`) and the container, ECS or EKS properties of its node range. Every node, main or child, has the job's own status, reason and times; real Batch starts the child nodes only once the main node is `RUNNING`, and a child node can finish or fail on its own. `CancelJob` and `TerminateJob` take the job ID, not a node ID. | ||
| - Fair-share scheduling policies are stored, validated and bound to queues, but they never change the order in which jobs run. Jobs never stay `RUNNABLE` because no capacity is free, so "Batch Job Queue Blocked" events are never sent. | ||
| - "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either. | ||
| - "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children and multi-node nodes send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either. | ||
| - Error messages are written by the emulator. The exception type (`ClientException`, HTTP 400) is the documented one. | ||
| - `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). Overrides are not checked against compute-environment platform rules (for example `nodeOverrides` on Fargate), and multi-node jobs report no per-node `nodeDetails`. | ||
| - `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). On a job queue whose compute environments are `FARGATE` or `FARGATE_SPOT`, `SubmitJob` rejects `nodeOverrides` and multi-node job definitions with a `ClientException`, and `RegisterJobDefinition` rejects `nodeProperties` with the `FARGATE` platform capability. Other platform rules are not checked: multi-node jobs on Spot or `UNMANAGED` compute environments, the `instanceType`, `memory` and `vcpus` container overrides on Fargate, and ECS versus EKS properties against the compute environment, since EKS compute environments are not modeled. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,246 @@ | ||
| package batch | ||
|
|
||
| import ( | ||
| "encoding/json" | ||
| "math" | ||
| "strconv" | ||
| "strings" | ||
|
|
||
| cerrors "github.com/stackshy/cloudemu/v2/errors" | ||
| "github.com/stackshy/cloudemu/v2/services/batch/driver" | ||
| ) | ||
|
|
||
| // nodeSeparator joins a multi-node parallel job's ID and a node index: node i | ||
| // of job J is "J#i" (Multi-node parallel jobs, Batch User Guide). | ||
| const nodeSeparator = "#" | ||
|
|
||
| // Compute resource types that run on Fargate (ComputeResource.type), and the | ||
| // job definition platform capability for Fargate. | ||
| const ( | ||
| crTypeFargate = "FARGATE" | ||
| crTypeFargateSpot = "FARGATE_SPOT" | ||
| platformFargate = "FARGATE" | ||
| ) | ||
|
|
||
| // nodeLayout is the node count, main node and node ranges of a multi-node | ||
| // parallel job, read from its nodeProperties (overrides already applied). | ||
| type nodeLayout struct { | ||
| numNodes int32 | ||
| mainNode int32 | ||
| ranges []jsonObject | ||
| } | ||
|
|
||
| // nodeLayoutWire decodes the part of nodeProperties a node layout needs. The | ||
| // int32 fields make an out-of-range numNodes or mainNode a decode error. | ||
| type nodeLayoutWire struct { | ||
| NumNodes int32 `json:"numNodes"` | ||
| MainNode int32 `json:"mainNode"` | ||
| NodeRangeProperties []jsonObject `json:"nodeRangeProperties"` | ||
| } | ||
|
|
||
| // parseNodeLayout reads a job's node layout. ok is false for a job that isn't | ||
| // a multi-node parallel job. | ||
| func parseNodeLayout(props json.RawMessage) (nodeLayout, bool) { | ||
| if !isSet(props) { | ||
| return nodeLayout{}, false | ||
| } | ||
|
|
||
| var w nodeLayoutWire | ||
| if json.Unmarshal(props, &w) != nil || w.NumNodes < 1 || w.MainNode < 0 || w.MainNode >= w.NumNodes { | ||
| return nodeLayout{}, false | ||
| } | ||
|
|
||
| return nodeLayout{numNodes: w.NumNodes, mainNode: w.MainNode, ranges: w.NodeRangeProperties}, true | ||
| } | ||
|
|
||
| // rangeFor returns the node range whose targetNodes covers node i, or nil. | ||
| func (l *nodeLayout) rangeFor(i int32) jsonObject { | ||
| for _, r := range l.ranges { | ||
| var target string | ||
| _ = json.Unmarshal(r["targetNodes"], &target) | ||
|
|
||
| nr, ok := parseTargetNodes(target, int(l.numNodes)) | ||
| if ok && int(i) >= nr.start && int(i) <= nr.end { | ||
| return r | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // splitNodeID splits "parent#index" into its parts. | ||
| func splitNodeID(id string) (parent string, idx int32, ok bool) { | ||
| i := strings.LastIndex(id, nodeSeparator) | ||
| if i <= 0 { | ||
| return "", 0, false | ||
| } | ||
|
|
||
| n, err := strconv.ParseInt(id[i+1:], 10, 32) | ||
| if err != nil || n < 0 { | ||
| return "", 0, false | ||
| } | ||
|
|
||
| return id[:i], int32(n), true | ||
| } | ||
|
|
||
| // lookupNode resolves a node job ID "<jobId>#<nodeIndex>" to its multi-node | ||
| // job's record, the node index and the job's node layout. | ||
| func (m *Mock) lookupNode(id string) (rec jobRecord, idx int32, layout nodeLayout, ok bool) { | ||
| parent, i, isNode := splitNodeID(id) | ||
| if !isNode { | ||
| return jobRecord{}, 0, nodeLayout{}, false | ||
| } | ||
|
|
||
| r, found := m.jobs.Get(parent) | ||
| if !found { | ||
| return jobRecord{}, 0, nodeLayout{}, false | ||
| } | ||
|
|
||
| l, isMNP := parseNodeLayout(r.Job.NodeProperties) | ||
| if !isMNP || i >= l.numNodes { | ||
| return jobRecord{}, 0, nodeLayout{}, false | ||
| } | ||
|
|
||
| return r, i, l, true | ||
| } | ||
|
|
||
| // renderNode builds node i of a multi-node parallel job. Nodes are not stored: | ||
| // each node follows the job's own lifecycle (status, reason, times and | ||
| // cancel/terminate flags), carries nodeDetails, and reports the container, | ||
| // ECS or EKS properties of the node range that covers it instead of the job's | ||
| // nodeProperties. | ||
| func (o *observer) renderNode(rec *jobRecord, l *nodeLayout, i int32) driver.Job { | ||
| j := o.render(rec, -1) | ||
|
|
||
| suffix := nodeSeparator + strconv.Itoa(int(i)) | ||
| j.JobID += suffix | ||
| j.JobARN += suffix | ||
| j.NodeProperties = nil | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SubmitJob accepts |
||
| j.NodeDetails = &driver.NodeDetails{NodeIndex: i, IsMainNode: i == l.mainNode, NumNodes: l.numNodes} | ||
|
|
||
| r := l.rangeFor(i) | ||
| j.Container = copyRaw(r["container"]) | ||
| j.EcsProperties = copyRaw(r["ecsProperties"]) | ||
| j.EksProperties = copyRaw(r["eksProperties"]) | ||
|
|
||
| if j.Status == driver.JobStatusSucceeded { | ||
| j.Container = withExitCode(j.Container) | ||
| } | ||
|
|
||
| return j | ||
| } | ||
|
|
||
| // nodes lists the nodes of a multi-node parallel job by index, resuming after | ||
| // the cursor index. Every node shares the job's name, definition, creation | ||
| // time, share identifier and status, so a status or filter that rejects one | ||
| // node rejects them all: the first node decides and the loop stays bounded by | ||
| // the page size. | ||
| func (l *jobLister) nodes(m *Mock) *driver.ListJobsOutput { | ||
| out := &driver.ListJobsOutput{Jobs: []driver.Job{}} | ||
|
|
||
| rec, ok := m.jobs.Get(l.in.MultiNodeJobID) | ||
| if !ok { | ||
| return out | ||
| } | ||
|
|
||
| layout, isMNP := parseNodeLayout(rec.Job.NodeProperties) | ||
|
|
||
| next := l.after + 1 | ||
| if !isMNP || next < 0 || next > math.MaxInt32 || next >= int64(layout.numNodes) { | ||
| return out | ||
| } | ||
|
|
||
| start := int32(next) | ||
|
|
||
| first := l.o.renderNode(&rec, &layout, start) | ||
| if !l.keeps(&first) { | ||
| return out | ||
| } | ||
|
|
||
| for i := start; i < layout.numNodes; i++ { | ||
| if len(out.Jobs) == l.limit { | ||
| out.NextToken = encodeListCursor(int64(i - 1)) | ||
|
|
||
| break | ||
| } | ||
|
|
||
| out.Jobs = append(out.Jobs, l.o.renderNode(&rec, &layout, i)) | ||
| } | ||
|
|
||
| return out | ||
| } | ||
|
|
||
| // keeps reports whether a listed job passes the status and filters. | ||
| func (l *jobLister) keeps(j *driver.Job) bool { | ||
| return (!l.useStatus || j.Status == l.status) && matchesFilters(j, l.in.Filters) | ||
| } | ||
|
|
||
| // fargateQueue reports whether a job queue runs its jobs on Fargate: its | ||
| // compute environments have FARGATE or FARGATE_SPOT compute resources. A queue | ||
| // can't mix Fargate and EC2 compute environments (Job queues on Fargate, Batch | ||
| // User Guide), so one Fargate compute environment decides. | ||
| func (m *Mock) fargateQueue(q *driver.JobQueue) bool { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This relies on a queue never mixing Fargate and EC2, but CreateJobQueue accepts |
||
| for _, o := range q.ComputeEnvironmentOrder { | ||
| ce, ok := m.computeEnvs.Get(nameFromARN(kindComputeEnvironment, o.ComputeEnvironment)) | ||
| if !ok || !isSet(ce.ComputeResources) { | ||
| continue | ||
| } | ||
|
|
||
| var cr struct { | ||
| Type string `json:"type"` | ||
| } | ||
|
|
||
| if json.Unmarshal(ce.ComputeResources, &cr) == nil && (cr.Type == crTypeFargate || cr.Type == crTypeFargateSpot) { | ||
| return true | ||
| } | ||
| } | ||
|
|
||
| return false | ||
| } | ||
|
|
||
| // validateJobPlatform rejects a submission the queue's platform can't run. On | ||
| // a Fargate queue, nodeOverrides isn't applicable (SubmitJob API reference: | ||
| // "use containerOverrides instead") and multi-node parallel jobs aren't | ||
| // supported (RegisterJobDefinition API reference, type and nodeProperties). | ||
| func (m *Mock) validateJobPlatform(q *driver.JobQueue, jd *driver.JobDefinition, in *driver.SubmitJobInput) error { | ||
| if !isSet(in.NodeOverrides) && !isMultiNode(jd) { | ||
| return nil | ||
| } | ||
|
|
||
| if !m.fargateQueue(q) { | ||
| return nil | ||
| } | ||
|
|
||
| if isSet(in.NodeOverrides) { | ||
| return cerrors.Newf(cerrors.InvalidArgument, | ||
| "nodeOverrides isn't applicable to jobs that run on Fargate resources (job queue %q uses Fargate "+ | ||
| "compute environments); use containerOverrides instead", q.Name) | ||
| } | ||
|
|
||
| return cerrors.Newf(cerrors.InvalidArgument, | ||
| "multi-node parallel job definition %q can't run on Fargate resources (job queue %q uses Fargate "+ | ||
| "compute environments)", jd.ARN, q.Name) | ||
| } | ||
|
|
||
| func isMultiNode(jd *driver.JobDefinition) bool { | ||
| return jd.Type == driver.JDTypeMultiNode || isSet(jd.NodeProperties) | ||
| } | ||
|
|
||
| // validateJobDefinitionPlatform enforces the RegisterJobDefinition rule that a | ||
| // job run on Fargate resources can't be multi-node: nodeProperties (and so | ||
| // type multinode) can't be combined with the FARGATE platform capability. | ||
| func validateJobDefinitionPlatform(in *driver.RegisterJobDefinitionInput) error { | ||
| if !isSet(in.NodeProperties) && in.Type != driver.JDTypeMultiNode { | ||
| return nil | ||
| } | ||
|
|
||
| for _, c := range in.PlatformCapabilities { | ||
| if c == platformFargate { | ||
| return cerrors.New(cerrors.InvalidArgument, | ||
| "multinode job definitions and nodeProperties aren't supported for jobs that run on Fargate "+ | ||
| "resources (platformCapabilities FARGATE)") | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ParseIntaccepts01and+1, sodescribe-jobs --jobs "$ID#01"returns a job whose jobId is$ID#1, which isn't the ID that was asked for. Real Batch treats a non-canonical node ID as unknown. Accept only0or digits with no leading zero, and apply the same check tosplitChildID.