RayJob lifecycle and image contract #42

Gate 2 of the tenant compute rollout — a human can run compute in their tenant. Verified by the parity test: one byte-identical RayJob manifest admitted under both the notebook and the API identity, and refused identically when broken.

Design: docs/design-doc/implementation/19-multi-tenancy.md (PR #674).

Home is provisional. This pack should be about the RayService CRD and model inference, which is useful well beyond the current deployment, and the RayJob machinery belongs elsewhere. The pack has been serving as a general Ray pack because RayService is built on RayCluster. Both items below are RayJob-shaped and should move when the pack is split; filing here now and moving later is cheaper than blocking on the split.

Problem

Two measured traps, both small, both blocking a second run.

Quota. ttlSecondsAfterFinished removes the RayCluster only. The RayJob object and the submitter Job hold count/rayjobs.ray.io and count/jobs.batch until something reaps them, so a tenant at a concurrency of two stops being able to submit after two runs.

Credentials. Neither stock rayproject/ray 2.41 nor 2.43 ships a botocore new enough for EKS Pod Identity — they carry 1.29.76, and the host 169.254.170.23 was only learned in 1.32.0. The driver gets no AWS credentials at all, and the failure reads as an IAM or trust-policy fault rather than an image fault. That is why it belongs in the image contract as a build assertion rather than a comment.

Work

• Enable RayJobDeletionPolicy, restating RayClusterStatusConditions=true because Helm replaces lists rather than merging them — it is an explicit false in the pack defaults today
• Or ship a reaper CronJob, if the deletion policy is unavailable at the pinned KubeRay version
• Assert botocore>=1.32 / boto3>=1.35 at build time, checking the installed package directly rather than trusting a version string
• Document the same requirement for bare notebook clusters via runtime_env
• Diagnostics survive the reap: the runner writes its stdout and traceback into the tenant's own bucket beside the result manifest, where the tenant's role already permits the write and the diagnostics inherit the tenant boundary. Give failed RayJobs a longer TTL than successful ones so there is a kubectl logs window as well

Acceptance

Quota returns to zero after a run completes with no manual deletion, a head pod in a tenant namespace obtains credentials, and a failed run leaves a readable traceback after its pods are gone.

_{🤖 claude-opus-5 (medium), claude-opus-5-5 (medium) · reviewed by @oldsj}