feat(hosts): hand a long run off a dsh-superagent worker (#964)

changelog.d/964-long-run-handoff.md

@@ -0,0 +1,6 @@
+Added
+`scripts/dsh-superagent-run.sh` hands a long evolution run off a
+dsh-superagent worker: it starts the run in a detached `tmux` session
+(`setsid` fallback), writes `waiting.json` with the run id, round and run
+directory, and returns at once, so a worker turn no longer holds its slot for
+the length of the run (#964).

docs/dsh-super-agent.md

@@ -118,6 +118,80 @@ Do not start a nested run from inside a run's turn. A worker that evolves a
 harness which itself starts runs multiplies cost with no ceiling — keep one run
 per Ticket, and make the second one a follow-up Ticket.
 
+### Handing a run off instead of holding the slot
+
+A worker's background shell job is killed at 30 minutes; an evolution run is
+often longer. Start it outside the worker's process group and give the slot
+back:
+
+```bash
+bash scripts/dsh-superagent-run.sh --store "$RUN_STORE" -- \
+    agentdescent --store "$RUN_STORE" evolve spec.json --detach \
+    --budget 5 --usd-per-call 0.01
+# waiting 3f9c1a… round 2 /runs/3f9c1a…
+# report /runs/3f9c1a…/waiting.json
+```
+
+`--budget` needs `--usd-per-call`: the engine counts calls, and a per-call price
+is the only way a dollar figure becomes a stop rule. With one and not the other
+the CLI exits 2 before creating anything (`budget_usd needs usd_per_call`), so
+the pair is what makes the run stop itself and write why.
+
+Two stores, one variable: the wrapper's `--store` is where the report goes, and
+the run's store is the CLI's own `--store`, so pass the same value on both. The
+wrapper cannot see the child's store — give it only to the wrapper and `evolve`
+writes to the default store, the wrapper waits out its deadline and reports
+`found: false`, and the read-back below cannot find the run.
+
+It starts the command in a detached `tmux` session (`setsid` when tmux is
+absent), waits for the run's own `status.json`, writes `waiting.json` beside it
+with `run_id`, `round`, `state`, `run_dir`, `driver`, `session` and `log`, and
+exits 0 within seconds. Post the run id, the round and the run directory on the
+Ticket and end the turn; nothing holds the slot.
+
+A store is reused for every Ticket, so it already holds earlier runs. The
+wrapper only reports a `status.json` that appears or changes after it launches,
+so a handoff into a used store reports `found: false` until its own run writes
+one, instead of naming a finished run from last week. The tmux session carries a
+per-launch suffix for the same reason: two handoffs into stores both called
+`runs` get two sessions, not one shared, refused name. One live handoff per
+store: two launched into one store inside the same wait window both read that
+store's `waiting.json` and the second can report the first's run, so give each
+handoff its own store or wait for the first to finish.
+
+Read it back from a follow-up Trigger, a person, or a scheduled check — the
+wrapper adds no daemon and no scheduler. `--store` is a **global** flag, so it
+goes before the verb: after the verb the CLI exits 2 with `unrecognized
+arguments`, and without it the run is looked up in the default store
+(`~/.agentdescent/runs`, or `$AGENTDESCENT_HOME/runs`), not in the handoff's.
+
+```bash
+agentdescent --store "$RUN_STORE" status "$RUN_ID" --brief   # one line; safe to poll
+agentdescent --store "$RUN_STORE" show "$RUN_ID"             # the artifact, once there is one
+```
+
+Measured against a handoff of a fake run, a second in — `status` keeps it to
+one line, and `show` reports the gap instead of inventing a result:
+
+```console
+$ agentdescent --store "$RUN_STORE" status fake-run-1 --brief
+fake-run-1  running   round 2      calls=0  skill_dir:./skill
+$ agentdescent --store "$RUN_STORE" show fake-run-1
+run_id: fake-run-1
+state: running
+note: no result yet
+```
+
+What the wrapper changes, and what it cannot:
+
+| failure | what happens |
+|---|---|
+| the worker's turn ends, or its background job is killed | with the `--detach` above, the run is in its own session — not the worker's process group, and not the wrapper's tmux session either, which holds only the launcher — so it keeps going when either ends |
+| no tmux on the host (`setsid` fallback) | survives the group kill, not a reboot; the run directory is still the handoff |
+| the host restarts | the run directory reads `failed`; `agentdescent resume <run_id>` continues it |
+| an orphan outlives the Ticket | `agentdescent --store "$RUN_STORE" cancel <run_id>` sends SIGTERM to the run's process group; `tmux kill-session -t <session>` only stops the wrapper's own launcher, because a `--detach` run is not in that session |
+| the budget runs out | the run stops itself and writes why into its status; the follow-up reads it, nobody is waiting |
+
 ## 7. The check
 
 ```bash

scripts/dsh-superagent-run.sh

@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Start a long evolution run outside the worker's process group, then hand it
+# back (#964). The worker reports `waiting` and ends its turn; a follow-up
+# Trigger, a person or a scheduled check reads the run with `status`/`show`.
+#
+#   dsh-superagent-run.sh --store DIR -- agentdescent --store DIR evolve spec.json --detach
+#
+# The wrapper's --store is where the report goes; the run's store is the CLI's
+# own --store. Give both the same value, or the run lands in the default store
+# and the report says found: false.
+#
+# tmux is preferred because dsh-superagent already depends on it and a session
+# is not in the worker's process group, so the kill that ends the worker's
+# background job does not reach the run. `setsid` is the fallback: it survives
+# that kill, not a reboot -- the run directory, not the process, is what a
+# worker hands on.
+#
+# Exit 0 once the run is detached and DIR/waiting.json is written. It never
+# waits for the run: a handoff that blocks is the failure this replaces.
+set -euo pipefail
+
+WAIT="${DSH_RUN_HANDOFF_WAIT:-10}"   # seconds to wait for the run's status.json
+
+die() { echo "error: $*" >&2; exit 2; }
+usage() { awk 'NR > 1 && /^#/ {sub(/^# ?/, ""); print; next} NR > 1 {exit}' "$0"; }
+
+STORE=""; CMD=()
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --store) [ $# -ge 2 ] || die "--store needs a directory"; STORE="$2"; shift 2 ;;
+    --) shift; CMD=("$@"); break ;;
+    -h|--help) usage; exit 0 ;;
+    *) die "unknown argument: $1 (try --help)" ;;
+  esac
+done
+[ -n "$STORE" ] || die "--store is required"
+[ ${#CMD[@]} -gt 0 ] || die "nothing to start: give -- <command ...>"
+mkdir -p "$STORE"
+STORE="$(cd "$STORE" && pwd)"
+command -v python3 >/dev/null || die "python3 is required to write the handoff report"
+
+LOG="$STORE/worker.log"
+# The store's basename is not unique -- a worker that hands off two runs keeps
+# both in a directory called `runs` -- so the session carries a per-launch
+# suffix. Two handoffs then never ask tmux for the same name (#1084 review).
+SESSION="ad-$(printf '%s' "$(basename "$STORE")" | tr -c 'A-Za-z0-9_.-' '-')-$$-$RANDOM"
+QUOTED="$(printf '%q ' "${CMD[@]}")"
+
+# What the store already holds, taken before the launch: a reused store always
+# has earlier runs in it, and reporting the newest of those as this handoff is
+# worse than reporting nothing (#1084 review). Only a status.json that this
+# launch creates or rewrites may be reported.
+SNAP="$STORE/.handoff-$$-$RANDOM.json"
+trap 'rm -f "$SNAP"' EXIT
+python3 - "$STORE" "$SNAP" <<'PY'
+import glob, json, os, sys
+store, out = sys.argv[1:3]
+snap = {}
+paths = [os.path.join(store, "status.json")] + sorted(
+    glob.glob(os.path.join(store, "*", "status.json")))
+for path in paths:
+    if os.path.isfile(path):
+        st = os.stat(path)
+        snap[path] = [st.st_mtime, st.st_size]
+with open(out, "w", encoding="utf-8") as fh:
+    json.dump(snap, fh)
+PY
+if command -v tmux >/dev/null 2>&1; then
+  DRIVER=tmux
+  tmux new-session -d -s "$SESSION" -c "$PWD" "$QUOTED >> '$LOG' 2>&1" \
+    || die "tmux refused the session $SESSION"
+elif command -v setsid >/dev/null 2>&1; then
+  DRIVER=setsid
+  setsid sh -c "$QUOTED >> '$LOG' 2>&1" >/dev/null 2>&1 &
+else
+  die "neither tmux nor setsid is available; this host cannot hand off a run"
+fi
+
+# The run owns its status.json (`<store>/<run_id>/status.json`, or the store
+# itself when the caller owns the run directory). Absent is reported as such:
+# an invented run id is worse than a gap a Trigger can see.
+python3 - "$STORE" "$WAIT" "$DRIVER" "$SESSION" "$LOG" "$QUOTED" "$SNAP" <<'PY'
+import glob, json, os, sys, time
+store, wait, driver, session, log, command, snap_path = sys.argv[1:8]
+before = json.load(open(snap_path, encoding="utf-8"))
+deadline, found = time.time() + float(wait), None
+while time.time() < deadline:
+    paths = [p for p in [os.path.join(store, "status.json")]
+             + sorted(glob.glob(os.path.join(store, "*", "status.json")))
+             if os.path.isfile(p)]
+    fresh = []
+    for path in paths:
+        try:
+            st = os.stat(path)
+        except OSError:
+            continue
+        seen = before.get(path)
+        if seen is None or [st.st_mtime, st.st_size] != seen:
+            fresh.append(path)
+    if fresh:
+        found = max(fresh, key=os.path.getmtime)
+        break
+    time.sleep(0.2)
+run_dir = os.path.dirname(found) if found else store
+status = json.load(open(found)) if found else {}
+report = {"status": "waiting", "found": bool(found),
+          "run_id": status.get("run_id", "") if found else "",
+          "round": status.get("round", 0), "state": status.get("state", "unknown"),
+          "run_dir": run_dir, "driver": driver,
+          "session": session if driver == "tmux" else "", "log": log,
+          "command": command.strip()}
+path = os.path.join(store, "waiting.json")
+with open(path, "w", encoding="utf-8") as fh:
+    json.dump(report, fh, indent=2, sort_keys=True)
+    fh.write("\n")
+print(f"waiting {report['run_id'] or '(no run yet)'} round {report['round']} {run_dir}")
+print(f"report {path}")
+PY