End local clusters when idle instead of at a fixed age

CLAUDE.md

@@ -1726,14 +1726,15 @@ Resolve the Slurm command user's UID through `id -u` on the same command runner,
 and use it for every native ownership check and filter.
 
 **Local compute needs no setup.** The built-in local offer provides detected usable
-logical CPUs and RAM, one node, fast startup, a 30-minute default and two-hour
-maximum lifetime. Loading the catalog writes no catalog and starts no cluster.
+logical CPUs and RAM, one node, fast startup, no walltime and a 30-minute idle
+timeout. Loading the catalog writes no catalog and starts no cluster.
 `lc compute launch` without CPU/memory flags selects only local offers and defaults
 the name to `local`. `--wait` returns when the accepted allocation is ready; timeout
 or startup failure retains its ID without resubmitting or terminating it.
 Configured remote offers precede the built-in local offer in selection order.
 Explicit local connections supply their own offers instead. `local.resources`
-overrides the built-in CPU/RAM budget and cannot accompany explicit local connections.
+and `local.time` override the built-in CPU/RAM budget and time limits and cannot
+accompany explicit local connections.
 `local.enabled: false` blocks local launch and execution while preserving inspection
 and termination. Recognized NERSC login nodes disable local compute automatically;
 other sites can disable it in their catalogs. Native permissions remain the
@@ -2134,6 +2135,34 @@ unlinks before writing; a new tampering test should too.
   override flag is needed. Local compute in interactive compute-node sessions
   remains available, subject to the configured local policy.
 
+- **Local compute ends when idle, not at a fixed age (2026-09, issue #233).**
+  An offer's `time` is `{default?, max?, idle?}` and needs a `default` or an
+  `idle`; the built-in local offer is `{idle: 30m}`, so a long recipe finishes
+  and the cluster stops 30 minutes after the last task. `idle` is handed to the
+  scheduler as Dask's own `idle_timeout` — its activity test (running, queued or
+  unrunnable tasks and any transition reset it; clients and `scheduler_info`
+  polls do not, measured) rather than a tracker of ours. `--time` stays a hard
+  walltime (SIGALRM, unconditional on activity); with both, the first to fire
+  ends the allocation, and the built-in offer has no `max` because a ceiling on
+  `--time` is meaningless when omitting it means unbounded. When the scheduler
+  closes without the owner asking (no SIGTERM yet), a `SchedulerPlugin.close`
+  hook records the reason in `error.json` and SIGKILLs the session itself, like
+  the walltime path — from the hook, so a scheduler that idles out before
+  `LocalCluster(...)` returns still ends the owner, and never through
+  `LocalCluster`'s own close, which waits ~34 s on the departed scheduler
+  (measured), holding the one-per-machine slot and the name. `SCHEDULER_CONFIG`
+  pins `idle-timeout: None`, so only an offer sets one — ambient Dask config
+  reaches neither local nor Slurm schedulers. `compute.connect` (execution only;
+  `status` connects through the provider) submits one no-op task, so a
+  driver's preparation — annex fetch, image build, sync — starts with a full
+  countdown. Slurm refuses `time.idle` and still needs `time.default`: its
+  allocations end at the native walltime, and an ignored idle timeout would be
+  a lie. Accepted residue: a preparation longer than the timeout still loses
+  the cluster; a driver pausing between tasks (a long annex commit) counts as
+  idle; and without `--time` nothing bounds an owner whose scheduler loop
+  wedges — the walltime's SIGALRM was that bound, and a second timer only for
+  it was judged not worth its code.
+
 - **Local compute accompanies remote catalogs (2026-09).** The built-in offer
   uses the host's usable CPU/RAM capacity and follows configured offers, replacing
   the previous one-CPU/1-GiB fallback that disappeared when a catalog existed.

docs/api/compute.md

@@ -13,16 +13,17 @@ It owns no service, registry, or saved current-cluster selection.
 | `Compute.discover()` | Snapshots and per-connection errors, querying each authority once. |
 | `Compute.status(cluster_id, wait=False, timeout=300)` | Resolve a name or full ID; return native allocation state plus authenticated Dask readiness. Waiting backs off from one to 30 seconds between native queries. |
 | `Compute.down(cluster_id)` | Resolve a name or full ID, request native termination independent of scheduler health, and return the canonical `Identity`. |
-| `connect(cluster_id, timeout=10)` | Resolve a name or full ID; borrow a standard Dask client, closing the client but never the allocation. |
+| `connect(cluster_id, timeout=10)` | Resolve a name or full ID; borrow a standard Dask client, closing the client but never the allocation. Submits one no-op task, so a caller's preparation restarts the idle countdown. |
 | `Provider` | `plan`, `launch`, `discover`, `inspect`, `connect`, `terminate`. |
 
 `Catalog.load()` defaults to `~/.lightcone/compute.yaml`. The built-in `local`
-offer uses detected usable CPUs and RAM, one node, fast startup, and a 30-minute
-default/two-hour maximum lifetime. `local.resources` overrides its CPU/RAM budget;
+offer uses detected usable CPUs and RAM, one node, fast startup, and no walltime:
+it ends after 30 minutes without task activity. `local.resources` overrides its
+CPU/RAM budget and `local.time` its time limits;
 `local.enabled: false` blocks local launch and execution while retaining connections
 for inspection and termination. Remote catalogs retain the implicit local offer
 unless disabled. Explicit local connections supply their own offers instead and
-cannot be combined with `local.resources`. GPU offers require explicit configuration.
+cannot be combined with `local.resources` or `local.time`. GPU offers require explicit configuration.
 Loading creates no configuration file or allocation.
 The effective local policy also disables local offers on recognized NERSC login
 nodes: nonempty `NERSC_HOST` and a short hostname matching `login[0-9]+`.
@@ -48,8 +49,12 @@ duplicate and non-string mapping keys before model validation. Provider-specific
 Units are explicit. `Resources.memory_gib` stores exact decimal GiB (the YAML key
 is `memory`), and `memory_bytes` derives an exact integer. Native observations use
 `Resources.from_bytes(...)`; requests store `Request.memory_bytes`. `TimeLimits`
-keeps the configured `default` and `max` duration strings and exposes
-`default_seconds` and `max_seconds`. `Startup.class_` corresponds to YAML `class`.
+keeps the configured `default`, `max`, and `idle` duration strings, each optional
+but requiring a `default` or an `idle`, and exposes `default_seconds`,
+`max_seconds`, and `idle_seconds` (`None` when unset). A `LaunchPlan` carries the
+resolved hard walltime as `seconds` and derives `idle_seconds` from its offer;
+either may be `None` for a local plan, while Slurm plans always have `seconds` and
+never `idle_seconds`. `Startup.class_` corresponds to YAML `class`.
 Connection names exist only as catalog mapping keys, referenced by `Offer.connection`.
 
 Compute memory accepts bare GiB quantities and SkyPilot-style binary units:
@@ -257,6 +262,7 @@ management; callers must respect the documented execution limits. Containers
 managed outside that process group can survive local teardown.
 
 Tests cover deterministic selection, malformed identities and catalogs, partial
-native failures, acceptance ambiguity, PID reuse, detached local lifetime, standard
+native failures, acceptance ambiguity, PID reuse, detached local walltime and idle
+expiry, standard
 Dask bootstrap, and explicit execution through borrowed clients. Slurm command
 contracts are simulated; a real NERSC submission remains a deployment check.

docs/cli/compute.md

@@ -13,16 +13,17 @@ lc compute down CLUSTER [--json]
 
 With no resource flags, `lc compute launch` starts the default local CPU offer,
 names the cluster `local`, and uses all detected usable logical CPUs and RAM.
-The built-in offer has one node, fast startup, a 30-minute default lifetime, and
-a two-hour maximum. `--name` and `--time` override the name and lifetime.
+The built-in offer has one node, fast startup, and no fixed lifetime: it ends
+after 30 minutes without task activity. `--name` overrides the name, and `--time`
+adds a hard lifetime that ends the cluster even while work is running.
 CPU and RAM are cooperative scheduling budgets, not exclusive reservations.
 GPUs still require explicit offers and, locally, a `CUDA_VISIBLE_DEVICES` mask.
 
 `~/.lightcone/compute.yaml` configures resource offers; `LC_COMPUTE_CONFIG` selects
 another file for all compute and execution commands. The top-level `local` block
-can override the built-in CPU/RAM budget or disable local compute. Without an
-explicit local connection, the built-in local offer is appended after configured
-offers. Catalogs with explicit local connections use their own offers instead;
+can override the built-in CPU/RAM budget or time limits, or disable local compute.
+Without an explicit local connection, the built-in local offer is appended after
+configured offers. Catalogs with explicit local connections use their own offers instead;
 the shortcut chooses the first eligible local offer. See
 [local configuration](../user/cluster.md#customize-resource-offers).
 Missing explicit paths and invalid catalogs are errors. Loading a catalog or
@@ -36,7 +37,7 @@ See [local allocations](../user/cluster.md#local-allocations) for detection deta
 
 | Command | Behavior |
 |---|---|
-| `resources` | Ordered available offers, per-node shape, node limit, default/maximum time, and startup class. Free capacity remains unknown. |
+| `resources` | Ordered available offers, per-node shape, node limit, default/maximum walltime, idle timeout, and startup class. Free capacity remains unknown. |
 | `launch` | Resolve one resource request and submit exactly once; print only the cluster name to stdout on acceptance. |
 | `launch --wait` | Submit once, then wait for all expected workers. `--timeout` sets the readiness deadline (default 300 seconds). |
 | `launch --dry-run` | Show the resolved shape and native launch parameters without allocation. |
@@ -83,11 +84,28 @@ maintain SkyPilot's accelerator alias registry: use the labels configured in
 
 Time accepts positive durations with day/hour/minute/second units, such as `30m`,
 `1h30m`, or `45s`. Without
-`--time`, the chosen offer's default applies. `fast` is a service class, not a
-queue-time promise. Limits apply to each allocation; aggregate quotas remain
-with the native backend.
+`--time`, the chosen offer's default walltime applies, if it has one. `fast` is a
+service class, not a queue-time promise. Limits apply to each allocation; aggregate
+quotas remain with the native backend.
+
+A local allocation ends at its walltime, after its idle timeout, or at whichever
+comes first when it has both. The idle timeout is Dask's scheduler
+`idle-timeout`: running or queued tasks keep the allocation alive, and new work
+restarts the countdown; connected clients and `status` queries do not. `lc run`
+and `lc materialize` restart it when they connect, so their preparation (fetching
+inputs, building the image, syncing the environment) starts with the full timeout.
+When it expires, the allocation ends, `status` gives that as the reason, and both
+its name and this machine's one local allocation are free again. A walltime ends
+the allocation even during active work. `down` still ends it at once.
+
+For Slurm, time is a finite native `--time` request, so a Slurm offer needs a
+`time.default` and cannot declare `time.idle`:
 
-For Slurm, time is a finite native `--time` request. Slurm's overtime and
+```text
+Error: Slurm allocations end at their native walltime: set the offer's time.default and remove time.idle
+```
+
+Slurm's overtime and
 termination-grace policy determines actual expiry and can allow unlimited
 overrun; Lightcone supplies no independent Slurm runtime deadline. A partition
 is passed only when explicitly set in the offer's configuration.

docs/user/cluster.md

@@ -9,8 +9,12 @@ present. `lc materialize --check` and `lc status` remain local project inspectio
 
 No configuration is needed on a fresh installation. `lc compute launch` uses all
 detected usable logical CPUs and RAM on this machine and names the cluster `local`.
-The default lifetime is 30 minutes, with a maximum of two hours. Use `--time` to
-change the lifetime. GPUs require explicit offers; see [GPU allocations](#gpu-allocations).
+A local cluster has no fixed lifetime: it ends after 30 minutes without task
+activity, so a two-hour recipe finishes normally and the cluster stops 30 minutes
+later if no further work arrives. Running or queued tasks keep it alive; connecting
+a client or checking its status does not. Use `--time` to add a hard lifetime, which
+ends the cluster even while work is running, or `lc compute down` to stop it now.
+GPUs require explicit offers; see [GPU allocations](#gpu-allocations).
 
 ```bash
 lc compute resources
@@ -45,8 +49,8 @@ Only one local cluster can run per user on each machine. A launch checks the
 process table for a running local cluster of yours and refuses if it finds one,
 including one launched through a different name, catalog, namespace, or
 connection root. End the existing cluster before launching another; once its
-owner process exits, including on failure or walltime expiry, a new launch
-proceeds. A refusal identifies the running cluster and its original catalog and
+owner process exits, including on failure, idle expiry, or walltime expiry, a
+new launch proceeds. A refusal identifies the running cluster and its original catalog and
 connection root. Use that catalog to inspect or stop the cluster if the current
 catalog no longer includes its connection. If the cluster's record is missing or
 damaged, the refusal names its process ID instead, to stop with `kill`.
@@ -55,9 +59,13 @@ visible where `lc` runs: a launch inside a container does not see a cluster
 started outside it.
 An allocation owns a detached process session and standard `LocalCluster`: one
 worker process with `task_slots_per_node` threads, and a scheduler that listens
-on `127.0.0.1` over TLS. Its own logs are discarded; a startup failure is kept
-and shown as the reason by `lc compute status`. At its time limit the whole
-process session is killed with SIGKILL, so a recipe still running stops mid-write.
+on `127.0.0.1` over TLS. Its own logs are discarded; a startup failure, or the
+scheduler closing after its idle timeout, is kept and shown as the reason by
+`lc compute status`. At its walltime the whole process session is killed with
+SIGKILL, so a recipe still running stops mid-write. The idle timeout ends the
+session the same way. Dask tracks tasks, not processes: after an interrupted
+`lc run` or `lc materialize`, a recipe can keep running once its task is gone,
+and the idle timeout stops it too.
 `down` sends SIGTERM, waits three seconds, then sends SIGKILL.
 Private process locators are checked against the native boot UUID, UID, process
 session, and exact command containing the allocation's random token before
@@ -128,6 +136,15 @@ Both CPU and memory are required in `local.resources`; omit that block to use
 detected capacity. The same capacity validation applies to configured budgets.
 This controls the default offer, not hard OS resource limits.
 
+`local.time` replaces the default offer's time limits, for example a longer idle
+timeout with a ceiling on `--time`:
+
+```yaml
+version: 1
+local:
+  time: {idle: 1h, max: 8h}
+```
+
 NERSC login nodes are guarded without setup. For other sites, or to disable local
 compute on every node using the catalog, set:
 
@@ -151,7 +168,7 @@ creates no configuration file.
 By default, a built-in `local` connection and offer accompany remote offers, with
 configured offers taking selection priority. If the catalog already defines local
 connections, those offers replace the implicit local offer; omit `local.resources`
-and size those offers directly. The no-resource shortcut selects the first eligible
+and `local.time`, and size and time those offers directly. The no-resource shortcut selects the first eligible
 local offer and defaults its cluster name to `local`.
 
 Resource requests can select the built-in local offer when no earlier remote
@@ -175,7 +192,7 @@ offers:
     connection: workstation
     resources: {cpus: 4, memory: 8}
     max_nodes: 1
-    time: {default: 30m, max: 2h}
+    time: {idle: 30m}
     startup: {class: fast}
 ```
 
@@ -196,8 +213,11 @@ and an ordered `offers` list. Connections and offers default to empty:
   optional `context`, and optional provider `launch` settings. Namespaces must
   be unique, and so must each provider/`context` pair.
 - An offer has a unique `name`, the `connection` it uses, per-node `resources`
-  (`cpus`, `memory`, and optional `accelerators`), `max_nodes`, and `time` with a
-  `default` no longer than its `max`. `startup` is optional (`fast`, `batch`, or the default
+  (`cpus`, `memory`, and optional `accelerators`), `max_nodes`, and `time`. `time`
+  holds an optional walltime `default`, no longer than an optional `max`, and an
+  optional `idle` timeout; it needs a `default` or an `idle`, so every allocation
+  can end. Local offers end at whichever limit comes first. Slurm offers need a
+  `default` and refuse `idle`. `startup` is optional (`fast`, `batch`, or the default
   `unknown`), written either as a bare class or as `{class: …, source: …}`.
   `config` holds provider-specific settings.
 
@@ -420,7 +440,7 @@ For local GPUs, add an offer to the [workstation catalog above](#customize-resou
     connection: workstation
     resources: {cpus: 4, memory: 8GB, accelerators: 'GPU:1'}
     max_nodes: 1
-    time: {default: 30m, max: 2h}
+    time: {idle: 30m}
     startup: fast
 ```
 
@@ -513,7 +533,7 @@ CUDA mask is empty. `lc run` reserves the worker's entire CPU, memory, and GPU
 budgets; direct and podman-hpc probes inherit that allocation mask.
 
 Recipe `time_limit` is not supported and is refused before preparation or
-execution. Set the allocation lifetime with `lc compute launch --time` instead.
+execution. Bound the allocation instead, with `lc compute launch --time`.
 Fractional CPU/GPU counts, GPU model requests inside a recipe, and disk requests
 are also rejected rather than ignored.
 

docs/user/getting-started.md

@@ -250,8 +250,8 @@ the record of what your results were computed with.
 ## 5. Materialize
 
 Launch the built-in local offer; no compute configuration is needed. It provides
-all usable CPUs and RAM for 30 minutes. Keep the returned ID in `CLUSTER` for this
-walkthrough. Configured remote offers coexist with that default. A catalog can
+all usable CPUs and RAM, and stops once it has had no work for 30 minutes. Keep
+the returned ID in `CLUSTER` for this walkthrough. Configured remote offers coexist with that default. A catalog can
 override the local budget, disable local compute, or provide explicit local offers;
 see [Running on a Cluster](cluster.md). NERSC login nodes automatically refuse local
 compute; use a compute node in an interactive allocation or a configured Slurm offer.

evals/prompt.md

@@ -53,9 +53,9 @@ inspect `lc compute status` and reuse it rather than launching another.
 existing cluster. Launch's `--wait` defaults to a 300-second readiness timeout;
 `--timeout SECONDS` overrides it. A waiting launch that fails reports the accepted
 cluster ID and leaves the allocation unchanged: inspect it before retrying.
-The default local lifetime is 30 minutes; `--time` overrides it up to two hours
-for the built-in offer. After expiry, launch again; the name `local` can be reused,
-but the immutable ID changes.
+A local cluster ends after 30 minutes without task activity; running work keeps
+it alive, and `--time` adds a hard lifetime that ends it even mid-run. After it
+ends, launch again; the name `local` can be reused, but the immutable ID changes.
 
 Compute configuration is `~/.lightcone/compute.yaml`, or the file selected by
 `LC_COMPUTE_CONFIG`. Its `local.resources` mapping can override the built-in CPU/RAM