Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/coverage/aws/batch.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ AWS's `batch` service · portable interface `driver.Batch` · [AWS index](./READ

- Jobs run no workload: a job always succeeds (container `exitCode` 0) once its dependencies succeed. By default it is `SUCCEEDED` as soon as it is submitted; with `--async-settle` it spends about one second each in `SUBMITTED`, `RUNNABLE` and `STARTING`, then five seconds in `RUNNING` (and stays `PENDING` while a dependency is unfinished). `attempts` is always empty, and retry strategies and timeouts are stored and echoed but never applied.
- `CancelJob` on a `PENDING` job takes effect at once. Real Batch waits for the job's dependencies to finish first.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. Multi-node parallel jobs run as a single job: `ListJobs` with `multiNodeJobId` returns an empty list.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished.
- Multi-node parallel jobs: nodes are derived from the job, not stored one by one. `DescribeJobs` on `<jobId>#<n>` and `ListJobs` with `multiNodeJobId` return each node with `nodeDetails` (`nodeIndex`, `isMainNode`) and the container, ECS or EKS properties of its node range. Every node, main or child, has the job's own status, reason and times; real Batch starts the child nodes only once the main node is `RUNNING`, and a child node can finish or fail on its own. `CancelJob` and `TerminateJob` take the job ID, not a node ID.
- Fair-share scheduling policies are stored, validated and bound to queues, but they never change the order in which jobs run. Jobs never stay `RUNNABLE` because no capacity is free, so "Batch Job Queue Blocked" events are never sent.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children and multi-node nodes send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- Error messages are written by the emulator. The exception type (`ClientException`, HTTP 400) is the documented one.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). Overrides are not checked against compute-environment platform rules (for example `nodeOverrides` on Fargate), and multi-node jobs report no per-node `nodeDetails`.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). On a job queue whose compute environments are `FARGATE` or `FARGATE_SPOT`, `SubmitJob` rejects `nodeOverrides` and multi-node job definitions with a `ClientException`, and `RegisterJobDefinition` rejects `nodeProperties` with the `FARGATE` platform capability. Other platform rules are not checked: multi-node jobs on Spot or `UNMANAGED` compute environments, the `instanceType`, `memory` and `vcpus` container overrides on Fargate, and ECS versus EKS properties against the compute environment, since EKS compute environments are not modeled.
7 changes: 4 additions & 3 deletions docs/coverage/nongoals/batch.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
- Jobs run no workload: a job always succeeds (container `exitCode` 0) once its dependencies succeed. By default it is `SUCCEEDED` as soon as it is submitted; with `--async-settle` it spends about one second each in `SUBMITTED`, `RUNNABLE` and `STARTING`, then five seconds in `RUNNING` (and stays `PENDING` while a dependency is unfinished). `attempts` is always empty, and retry strategies and timeouts are stored and echoed but never applied.
- `CancelJob` on a `PENDING` job takes effect at once. Real Batch waits for the job's dependencies to finish first.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished. Multi-node parallel jobs run as a single job: `ListJobs` with `multiNodeJobId` returns an empty list.
- Array jobs: children are derived from the parent, not stored one by one. `SEQUENTIAL` and `N_TO_N` dependencies are honoured. An array parent reports `PENDING` while any child is unfinished.
- Multi-node parallel jobs: nodes are derived from the job, not stored one by one. `DescribeJobs` on `<jobId>#<n>` and `ListJobs` with `multiNodeJobId` return each node with `nodeDetails` (`nodeIndex`, `isMainNode`) and the container, ECS or EKS properties of its node range. Every node, main or child, has the job's own status, reason and times; real Batch starts the child nodes only once the main node is `RUNNING`, and a child node can finish or fail on its own. `CancelJob` and `TerminateJob` take the job ID, not a node ID.
- Fair-share scheduling policies are stored, validated and bound to queues, but they never change the order in which jobs run. Jobs never stay `RUNNABLE` because no capacity is free, so "Batch Job Queue Blocked" events are never sent.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- "Batch Job State Change" events go to the EventBridge default bus on the serve scheduler's tick (and on any Batch job API call), one per status the job passed through; none is sent for the initial submission. In library mode, nothing ticks, so events are published when a job API call sees the change or when `Tick` is called. A job that fails reports only `FAILED`, without the earlier steps. Array children and multi-node nodes send no events of their own. Real Batch sends no state-change events for compute environments or job queues, so the emulator sends none either.
- Error messages are written by the emulator. The exception type (`ClientException`, HTTP 400) is the documented one.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). Overrides are not checked against compute-environment platform rules (for example `nodeOverrides` on Fargate), and multi-node jobs report no per-node `nodeDetails`.
- `containerOverrides`, `ecsPropertiesOverride`, `eksPropertiesOverride` and `nodeOverrides` are merged into the job detail `DescribeJobs` returns (environment variables by name, resource requirements by type, other fields replaced). On a job queue whose compute environments are `FARGATE` or `FARGATE_SPOT`, `SubmitJob` rejects `nodeOverrides` and multi-node job definitions with a `ClientException`, and `RegisterJobDefinition` rejects `nodeProperties` with the `FARGATE` platform capability. Other platform rules are not checked: multi-node jobs on Spot or `UNMANAGED` compute environments, the `instanceType`, `memory` and `vcpus` container overrides on Fargate, and ECS versus EKS properties against the compute environment, since EKS compute environments are not modeled.
4 changes: 4 additions & 0 deletions providers/aws/batch/job_definition.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ func (m *Mock) RegisterJobDefinition(
return nil, err
}

if err := validateJobDefinitionPlatform(&in); err != nil {
return nil, err
}

m.registerMu.Lock()

cur, _ := m.jobDefMaxRev.Get(in.Name)
Expand Down
5 changes: 5 additions & 0 deletions providers/aws/batch/job_lifecycle.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,11 @@ func cloneJob(j driver.Job) driver.Job {
j.ArrayProperties = &ap
}

if j.NodeDetails != nil {
nd := *j.NodeDetails
j.NodeDetails = &nd
}

j.Parameters = copyTags(j.Parameters)
j.Container = copyRaw(j.Container)
j.NodeProperties = copyRaw(j.NodeProperties)
Expand Down
246 changes: 246 additions & 0 deletions providers/aws/batch/job_nodes.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
package batch

import (
"encoding/json"
"math"
"strconv"
"strings"

cerrors "github.com/stackshy/cloudemu/v2/errors"
"github.com/stackshy/cloudemu/v2/services/batch/driver"
)

// nodeSeparator joins a multi-node parallel job's ID and a node index: node i
// of job J is "J#i" (Multi-node parallel jobs, Batch User Guide).
const nodeSeparator = "#"

// Compute resource types that run on Fargate (ComputeResource.type), and the
// job definition platform capability for Fargate.
const (
crTypeFargate = "FARGATE"
crTypeFargateSpot = "FARGATE_SPOT"
platformFargate = "FARGATE"
)

// nodeLayout is the node count, main node and node ranges of a multi-node
// parallel job, read from its nodeProperties (overrides already applied).
type nodeLayout struct {
numNodes int32
mainNode int32
ranges []jsonObject
}

// nodeLayoutWire decodes the part of nodeProperties a node layout needs. The
// int32 fields make an out-of-range numNodes or mainNode a decode error.
type nodeLayoutWire struct {
NumNodes int32 `json:"numNodes"`
MainNode int32 `json:"mainNode"`
NodeRangeProperties []jsonObject `json:"nodeRangeProperties"`
}

// parseNodeLayout reads a job's node layout. ok is false for a job that isn't
// a multi-node parallel job.
func parseNodeLayout(props json.RawMessage) (nodeLayout, bool) {
if !isSet(props) {
return nodeLayout{}, false
}

var w nodeLayoutWire
if json.Unmarshal(props, &w) != nil || w.NumNodes < 1 || w.MainNode < 0 || w.MainNode >= w.NumNodes {
return nodeLayout{}, false
}

return nodeLayout{numNodes: w.NumNodes, mainNode: w.MainNode, ranges: w.NodeRangeProperties}, true
}

// rangeFor returns the node range whose targetNodes covers node i, or nil.
func (l *nodeLayout) rangeFor(i int32) jsonObject {
for _, r := range l.ranges {
var target string
_ = json.Unmarshal(r["targetNodes"], &target)

nr, ok := parseTargetNodes(target, int(l.numNodes))
if ok && int(i) >= nr.start && int(i) <= nr.end {
return r
}
}

return nil
}

// splitNodeID splits "parent#index" into its parts.
func splitNodeID(id string) (parent string, idx int32, ok bool) {
i := strings.LastIndex(id, nodeSeparator)
if i <= 0 {
return "", 0, false
}

n, err := strconv.ParseInt(id[i+1:], 10, 32)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ParseInt accepts 01 and +1, so describe-jobs --jobs "$ID#01" returns a job whose jobId is $ID#1, which isn't the ID that was asked for. Real Batch treats a non-canonical node ID as unknown. Accept only 0 or digits with no leading zero, and apply the same check to splitChildID.

if err != nil || n < 0 {
return "", 0, false
}

return id[:i], int32(n), true
}

// lookupNode resolves a node job ID "<jobId>#<nodeIndex>" to its multi-node
// job's record, the node index and the job's node layout.
func (m *Mock) lookupNode(id string) (rec jobRecord, idx int32, layout nodeLayout, ok bool) {
parent, i, isNode := splitNodeID(id)
if !isNode {
return jobRecord{}, 0, nodeLayout{}, false
}

r, found := m.jobs.Get(parent)
if !found {
return jobRecord{}, 0, nodeLayout{}, false
}

l, isMNP := parseNodeLayout(r.Job.NodeProperties)
if !isMNP || i >= l.numNodes {
return jobRecord{}, 0, nodeLayout{}, false
}

return r, i, l, true
}

// renderNode builds node i of a multi-node parallel job. Nodes are not stored:
// each node follows the job's own lifecycle (status, reason, times and
// cancel/terminate flags), carries nodeDetails, and reports the container,
// ECS or EKS properties of the node range that covers it instead of the job's
// nodeProperties.
func (o *observer) renderNode(rec *jobRecord, l *nodeLayout, i int32) driver.Job {
j := o.render(rec, -1)

suffix := nodeSeparator + strconv.Itoa(int(i))
j.JobID += suffix
j.JobARN += suffix
j.NodeProperties = nil

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SubmitJob accepts arrayProperties with a multinode definition. When it does, each J#i node inherits the parent's arrayProperties (size and statusSummary) from render(rec, -1). I'd clear j.ArrayProperties here as well, since a node isn't an array parent.

j.NodeDetails = &driver.NodeDetails{NodeIndex: i, IsMainNode: i == l.mainNode, NumNodes: l.numNodes}

r := l.rangeFor(i)
j.Container = copyRaw(r["container"])
j.EcsProperties = copyRaw(r["ecsProperties"])
j.EksProperties = copyRaw(r["eksProperties"])

if j.Status == driver.JobStatusSucceeded {
j.Container = withExitCode(j.Container)
}

return j
}

// nodes lists the nodes of a multi-node parallel job by index, resuming after
// the cursor index. Every node shares the job's name, definition, creation
// time, share identifier and status, so a status or filter that rejects one
// node rejects them all: the first node decides and the loop stays bounded by
// the page size.
func (l *jobLister) nodes(m *Mock) *driver.ListJobsOutput {
out := &driver.ListJobsOutput{Jobs: []driver.Job{}}

rec, ok := m.jobs.Get(l.in.MultiNodeJobID)
if !ok {
return out
}

layout, isMNP := parseNodeLayout(rec.Job.NodeProperties)

next := l.after + 1
if !isMNP || next < 0 || next > math.MaxInt32 || next >= int64(layout.numNodes) {
return out
}

start := int32(next)

first := l.o.renderNode(&rec, &layout, start)
if !l.keeps(&first) {
return out
}

for i := start; i < layout.numNodes; i++ {
if len(out.Jobs) == l.limit {
out.NextToken = encodeListCursor(int64(i - 1))

break
}

out.Jobs = append(out.Jobs, l.o.renderNode(&rec, &layout, i))
}

return out
}

// keeps reports whether a listed job passes the status and filters.
func (l *jobLister) keeps(j *driver.Job) bool {
return (!l.useStatus || j.Status == l.status) && matchesFilters(j, l.in.Filters)
}

// fargateQueue reports whether a job queue runs its jobs on Fargate: its
// compute environments have FARGATE or FARGATE_SPOT compute resources. A queue
// can't mix Fargate and EC2 compute environments (Job queues on Fargate, Batch
// User Guide), so one Fargate compute environment decides.
func (m *Mock) fargateQueue(q *driver.JobQueue) bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relies on a queue never mixing Fargate and EC2, but CreateJobQueue accepts order=1,ce-ec2 order=2,ce-fg today, and on that queue every MNP submission is rejected. That's a CreateJobQueue/UpdateJobQueue gap rather than one in this PR. A follow-up that rejects the mixed order would make this check hold.

for _, o := range q.ComputeEnvironmentOrder {
ce, ok := m.computeEnvs.Get(nameFromARN(kindComputeEnvironment, o.ComputeEnvironment))
if !ok || !isSet(ce.ComputeResources) {
continue
}

var cr struct {
Type string `json:"type"`
}

if json.Unmarshal(ce.ComputeResources, &cr) == nil && (cr.Type == crTypeFargate || cr.Type == crTypeFargateSpot) {
return true
}
}

return false
}

// validateJobPlatform rejects a submission the queue's platform can't run. On
// a Fargate queue, nodeOverrides isn't applicable (SubmitJob API reference:
// "use containerOverrides instead") and multi-node parallel jobs aren't
// supported (RegisterJobDefinition API reference, type and nodeProperties).
func (m *Mock) validateJobPlatform(q *driver.JobQueue, jd *driver.JobDefinition, in *driver.SubmitJobInput) error {
if !isSet(in.NodeOverrides) && !isMultiNode(jd) {
return nil
}

if !m.fargateQueue(q) {
return nil
}

if isSet(in.NodeOverrides) {
return cerrors.Newf(cerrors.InvalidArgument,
"nodeOverrides isn't applicable to jobs that run on Fargate resources (job queue %q uses Fargate "+
"compute environments); use containerOverrides instead", q.Name)
}

return cerrors.Newf(cerrors.InvalidArgument,
"multi-node parallel job definition %q can't run on Fargate resources (job queue %q uses Fargate "+
"compute environments)", jd.ARN, q.Name)
}

func isMultiNode(jd *driver.JobDefinition) bool {
return jd.Type == driver.JDTypeMultiNode || isSet(jd.NodeProperties)
}

// validateJobDefinitionPlatform enforces the RegisterJobDefinition rule that a
// job run on Fargate resources can't be multi-node: nodeProperties (and so
// type multinode) can't be combined with the FARGATE platform capability.
func validateJobDefinitionPlatform(in *driver.RegisterJobDefinitionInput) error {
if !isSet(in.NodeProperties) && in.Type != driver.JDTypeMultiNode {
return nil
}

for _, c := range in.PlatformCapabilities {
if c == platformFargate {
return cerrors.New(cerrors.InvalidArgument,
"multinode job definitions and nodeProperties aren't supported for jobs that run on Fargate "+
"resources (platformCapabilities FARGATE)")
}
}

return nil
}
Loading
Loading