Honor recipe resources and allocate GPUs with SkyPilot-style requests

[EiffL]

Recipes now retain CPU, memory, and GPU requirements through planning and execution. Materialization checks that outputs which may rebuild fit a worker, then submits ordinary Dask tasks with resource reservations. Already-current outputs become dependency values without a resource reservation, and omitted recipe memory no longer serializes otherwise independent tasks. Worker capacities are parsed once per invocation.

Cluster requests use SkyPilot-style notation: --cpus 8+ --memory 32GB+ --gpus A100:4, with catalog accelerators: A100:4 or {A100: 4}. CPU and memory support exact/minimum requests; allocation GPU type/count are exact, while GPU:N accepts any configured model. Slurm translates these into native per-node GRES and preserves its CUDA device mask. GPU fields default to zero for CPU allocations.

GPU recipes require at least their declared GPU count, reserve the worker's full GPU budget, and inherit its whole mask. CPU recipes expose no GPUs. The built-in local offer remains CPU-only; local GPU offers require explicit configuration and a nonempty CUDA_VISIBLE_DEVICES on Linux. Configured local GPU counts and models are not hardware-verified or exclusively reserved. There is no CUDA inventory, device-assignment service, or custom Dask worker.

lc run reserves a whole worker. Direct and podman-hpc probes can use its GPUs; ordinary Docker/Podman probes run CPU-only with a diagnostic note. Explicit GPU recipes require direct execution or podman-hpc and reject unsupported container runtimes before image preparation. The host initializes NVIDIA device nodes; standalone GPU reruns require an explicit device mask.

Reservations coordinate scheduling, not per-command OS limits or numerical-library thread pools. Recipe time_limit remains unsupported; allocation walltime works. Compute memory follows SkyPilot's binary units; recipe memory preserves ASTRA units. Dask's normal scheduling and recomputation behavior is unchanged.

Validation: all 1,095 local tests, Ruff, clean-cache mypy, and the documentation build pass. All automated GitHub checks pass at 1b614ca: Linux/Python 3.11–3.13, macOS/Python 3.13, lint, agent evaluation, automated review, and sign-off validation. The external DCO gate awaits PR approval. Physical GPU execution and a real Slurm GPU allocation still need site validation.

[github-actions]

✅ Eval

Metric	Value
Outputs check	success
Agent run	success
Turns	7
Tool calls	5
Cost	$0.12
Agent wall time	0m45s
Model	claude-sonnet-5-5

lc status

  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)
  crate:   up to date with the outputs

  · current  baseline/best_fit        db4514a
  · current  baseline/hubble_diagram  db4514a
  · current  baseline/residuals       db4514a

3 current

Confusion & pain points (Claude analysis)

Confusion & pain points

• The run was essentially clean. There were no errored tool calls, and the agent finished the build in about 45 seconds. What friction there was came from the agent working blind to the CLI surface, not from failures.
• The agent guessed at the lc compute surface. It launched with lc compute launch --cpus 1 --memory 1 --name analysis and never ran --help or lc compute resources first. The guess worked, but a mistyped memory unit or flag would have cost a round trip. The eval prompt or the astra skill could show one canonical launch → materialize <name> → down sequence. The trace shows no hint of the mandatory cluster-name positional argument.
• The agent pinned the validator with uvx astra-tools@0.2.18 validate. It used this instead of an astra binary on PATH, which suggests the harness doesn't provide astra directly and the agent knew the version out of band. A version pin the agent must remember is fragile. The environment should expose astra, or the skill should name the exact invocation.
• The agent rewrote astra.yaml with an inline Python str.replace script. It matched literal recipe:\n command: python scripts/... blocks, so it depended on the scaffold's exact whitespace. It would break silently on any layout drift, and it wasn't checked with a diff before validating. It hints that there is no astra edit or scaffold command for adding format:, inputs: and decisions: to an existing output. A supported edit verb, or a skill example showing the complete output stanza, would remove the need for text surgery.
• The agent picked a license on its own. It ran a sed that inserted license = "CC-BY-4.0" because a license is what turns on crate maintenance. Nothing in the trace shows the task asking for one. The agent mentioned the choice in its summary, so it was disclosed, but the docs or task should say whether publication intent is expected. Otherwise an agent will invent licensing terms for someone's data to get ro-crate-metadata.json.
• The agent checked convergence in one pass, with no failure to learn from. It did a re-materialize, --check, a crate check and a clean-tree check at the end, so it knew the expected exit semantics. The final summary noted that it hadn't checked how many points the redshift cut removes. This is a science-verification gap, not a tooling one.

Full trace: agent-trace artifact on this run.

[EiffL]

Code review of main...fd9b2d3 (15 findings, inline below, most severe first).

The full test suite passes (1067 tests). test_allocation_survives_launcher_and_borrowed_client_exit failed once under -x but passed alone and in the full run, so it looks like a timing flake under load.

The four that change behavior most:

• Current outputs still take a resource reservation. Skips are validated and reserved too, so a CPU-only cluster cannot run a project containing any GPU output, even an already-current one.
• An undeclared memory reserves a worker's whole memory budget. No existing project declares memory, so every one drops to one task per worker.
• lc run on a GPU cluster always requests GPUs. In containerized mode under docker or podman, every probe is refused, CPU-only ones included.
• The GPU container runtime is checked per task, not up front. A docker/podman project with GPU recipes builds and commits the image before its GPU tasks fail.

🤖 Generated with Claude Code

[EiffL]

Skipped current outputs still reserve resources

Every task in the graph is validated and submitted with a resource reservation, including tasks the worker will only classify as current and skip; the fit check and the reservation both apply to skips.

Failure scenario: A project with one GPU output that is already current, plus a CPU plot to remake: lc materialize <cpu-cluster> fails in validate ("no worker in this cluster can satisfy that request") and remakes nothing. On a GPU cluster the skip also holds the worker's whole GPU budget, and, when memory is undeclared, its whole MEMORY budget, so a run with nothing to do checks its N current outputs one at a time per worker instead of in parallel across task slots.

[EiffL]

Undeclared memory reserves the whole worker budget

A recipe with no memory declaration reserves the worker's entire MEMORY budget. Every existing project declares none, so concurrency drops from task_slots_per_node to one task per worker. CLAUDE.md still says "each node's worker runs task_slots_per_node tasks at once".

Failure scenario: A 128-CPU Slurm node with 127 task slots running a multiverse spec that declares no resources previously ran ~127 recipes at once; now each worker runs them one at a time, a large slowdown for unchanged projects. The layer-4 invariant in CLAUDE.md now contradicts the code.

[EiffL]

lc run on a GPU cluster refuses under docker/podman

The probe always sets use_gpus when the worker advertises any GPUs. In containerized mode, policy_for then calls require_gpu_runtime, so every lc run on a GPU allocation refuses under docker or podman, even for CPU-only commands.

Failure scenario: A containerized project using podman, a local GPU allocation (CUDA_VISIBLE_DEVICES=0): lc run $C -- python -c 'print(1)' fails with "GPU containers require podman-hpc". The CLI offers no way to probe that cluster without GPUs.

[EiffL]

GPU container-runtime check runs per task, after the build

Whether the container runtime supports GPUs is checked per task in the worker, not in this up-front validate. So a docker or podman project with GPU recipes first builds the image (runtime_for_run(build=True), line 525) and converges the environment, then each GPU task fails.

Failure scenario: A docker project with a gpus: 1 recipe on a GPU cluster: validate passes, runtime_for_run(build=True) builds and commits an archive, CPU tasks run, and each GPU task then fails with "GPU containers require podman-hpc". This contradicts the documented "whole selected graph is checked before preparation".

[EiffL]

Local Nanny pins BLAS/OpenMP threads to 1

LocalCluster(processes=True) spawns its worker through a Nanny, whose default pre-spawn-environ sets OMP_NUM_THREADS, MKL_NUM_THREADS and OPENBLAS_NUM_THREADS to 1. Recipes inherit that through child_env(), so a cpus: 8 reservation still runs BLAS/OpenMP single-threaded locally, while Slurm workers (in-process Worker, no Nanny) are unrestricted.

Failure scenario: A recipe declaring cpus: 8 doing numpy linear algebra reserves 8 CPUs on a local cluster but uses 1 thread (8x slower than expected). The same recipe on Slurm uses every thread on the node, so the reservation means different things on the two providers.

[EiffL]

Test leaks CUDA_DEVICE_ORDER into the session

test_gpu_worker_advertises_native_capacity_with_the_native_mask runs slurm_bootstrap.run with gpus=2. _allocation then writes os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID', which the test never monkeypatches, so the value leaks into the rest of the session. The root cause is a validation function that mutates the process environment.

Failure scenario: Later tests that read CUDA_DEVICE_ORDER, such as local plan freezing or exec_policy(use_gpus=True) overlays, see PCI_BUS_ID, and whether they pass depends on test order.

[EiffL]

Request.gpus / accelerator_name are dead code

Request.gpus and Request.accelerator_name are called only by tests/test_compute.py:422; no production code uses them.

Failure scenario: Breaks the CLAUDE.md rule "No dead code. If nothing in the current layer calls it, it doesn't land yet." They also duplicate the identical properties on Resources.

[EiffL]

GPU-runtime refusal is enforced twice

The GPU-runtime refusal is enforced twice: in container.policy_for (container.py:434, via a lazy import of oci) and again here in OCIBackend.wrap.

Failure scenario: Two copies of one rule, reading different inputs (runtime.runtime versus self.runtime). A later change to one, such as adding a runtime, can leave the two disagreeing. Keep the check in one place, the backend or policy_for.

[EiffL]

validate re-parses worker capacity for every task

validate calls TaskResources.parse(...).requirements(self.workers) once per task, and each call re-scans and re-validates every worker's resource dict.

Failure scenario: The work is O(tasks × workers). A multiverse graph with thousands of tasks on a many-node allocation repeats the same capacity parsing thousands of times. Computing the capacity set once and matching each task against it would be enough.

[EiffL]

Three exception-conversion layers for one regex

duration_seconds has a single caller, model.duration, and the Pydantic path reaches it through three layers: model._duration converts ComputeError to ValueError, around model.duration, which converts ValueError to ComputeError, around units.duration_seconds.

Failure scenario: Three layers of exception conversion for one regex add indirection with no second caller of the inner function. This goes against CLAUDE.md's "Streamline before shipping. No small helper functions … where a few inline lines read fine."