Repository navigation
feat(hosts): hand a long run off a dsh-superagent worker (#964) - #1084
Conversation
|
Review named at Frame is settled on my side: the four What is wanted at this tip:
One body fix is already ordered and does not change the head: the |
|
Advisory review at Ran (my own runs; no CI leg used as evidence):
Two blocking items, both in the new §6 and both user-facing:
Non-blocking, each with a repro:
|
|
Reviewer's findings check out against the tip, and I re-read each one myself rather than taking the report on trust — Ruling: fix it in this PR, one push on
Everything else in the body stays. Name the new sha here once; Reviewer re-reads at that tip and I take it from there. |
|
Verifier report (advisory — not a counting PASS) Tip SHA: Ran:
The gated smoke, run four times at the tip: 2 passed, 2 failed, every failure the same assertion — Docs: section 6 of Not run:
What this leaves open, plainly: acceptance row 4 ("leaves no orphan") is not deterministic as written — it failed 2 of 4 runs on this host, and because the smoke is skipped by default CI will never see it. The wrapper looked right in both failing runs (session gone immediately after); the check needs to poll Advisory only: no PASS, no merge, no push. |
|
Test's run at That lands inside item 3 and does not change the ruling: the process-group-kill evidence in the body is the stronger claim, and the test has to earn it deterministically.
If the poll fix does not give 4/4, say so in the body and change The other five items from my previous comment stand unchanged, including dropping the |
The read-back command as written read the default store and exited 2, and the table said a --detach run lives in the wrapper's tmux session. It does not: runstore.launch starts it in its own session, so the session holds only the launcher and the orphan remedy is 'agentdescent --store ... cancel <run_id>', which signals the run's process group.
`tmux has-session -t ad-runs` is a prefix target, so the check could answer about someone else's session: with an unrelated `ad-runs-zzz` on the server it returned 0, and the smoke would then wait on, or report, a session the wrapper never started. `=` asks for that exact name. Local experiment only; nothing pushed.
The run writes status.json state=finished before its own process exits, and tmux destroys the session after the pane's command ends, so has-session can still return 0 for a moment: 3/12 replicas of this check read a live run (or a session mid-teardown) as an orphan, and 1/12 still did with a 1s sleep, so the window is longer than a sleep can cover -- 1.7s of I/O wait in one sample. Poll until the session goes, 15s deadline, and put the session name, the wait and the last reading in the failure message so a real orphan is diagnosable. Local experiment only; nothing pushed.
|
Verdict: CHANGES REQUESTED Tip SHA: Ran:
|
|
Verifier report (advisory — not a counting PASS; Tip SHA: Ran:
Not run: full Findings Blocking
Non-blocking
Risk / merge: T3 — |
Three review findings at b05c3f9. The page gave --store to the wrapper only, so `evolve` wrote to the default store and the report said found: false; the example and the wrapper's own usage now name the CLI's store too, and the page says why the wrapper cannot do it itself. A reused store lent the report its newest earlier run, so the wrapper snapshots the store before launching and accepts only a status.json that appears or changes after that. The tmux session name came from the store's basename, so two handoffs into stores both called `runs` collided; it now carries a per-launch suffix.
|
Record for the tip under review here: The last verdict on this PR names
The two advisory reports ( At
|
|
Verifier report (advisory — not a counting PASS; Review tip: This is the tip after Birfy's CHANGES REQUESTED at Ran:
Not run: full Findings Blocking
Non-blocking
Risk / merge: T3 — |
|
Verdict: PASS Tip SHA: Ran:
Not run: full This is the tip after CHANGES REQUESTED at
|
…1084) The page's example ran `evolve spec.json --detach --budget 5` with no price per call, and `runstore.require_budget_pair` refuses that pair: the CLI exits 2 before creating anything, so the documented command could not start a run. The test that was supposed to catch it asserted a substring, so a command that exits 2 passed. It now parses the fenced block, hands the child's argv to the CLI's own parser, and requires the documented pair to satisfy `require_budget_pair` -- so the page and the CLI cannot drift apart again. Also records the measured expectation that one store carries one live handoff: two launched inside the same wait window both read that store's waiting.json.
Partial: #964
What changed
A worker's background shell job is killed at 30 minutes; an evolution run is
often longer. The two options were blocking the slot for the length of the run,
or losing it.
scripts/dsh-superagent-run.sh(new) — starts the command in a detachedtmuxsession (setsidwhen tmux is absent, so either way it leaves theworker's process group), waits for the run's own
status.json, writeswaiting.jsonbeside it withrun_id,round,state,run_dir,driver,sessionandlog, and returns. Against a fake 4 s run it returned in0.065 s and the run was still alive.
docs/dsh-super-agent.md§6 — the pattern: start detached, reportwaitingwith the run id and round, follow up with
status/show; plus a table of thefour failures the wrapper cannot remove (no tmux, host restart, an orphan,
budget exhaustion) and
resumefor the restart case.tests/test_dsh_superagent_run.py(new) — an offline row that the page'scommand matches the wrapper and that
evolve --detachstill exists, and agated smoke (
AGENTDESCENT_LONG_RUN_SMOKE=1) that the caller returns whilethe run continues and that no tmux session is left once it finishes.
No daemon, no scheduler, no new run store: the run directory is the handoff, and
the worker's turn ends as soon as
waiting.jsonexists.Acceptance rows at the tip
Measured at
b05c3f9d97ac37b9e3730c8a2d34cdf42710aa4f; the later commits changethe page's example, which
status.jsonthe report accepts and the tmux sessionname, not what these four rows measure: the detach path they exercise (the
tmux/
setsidlaunch, thewaiting.jsonfields, the process surviving thecaller) is unchanged since
43155caa, so every command below still applies.status/showos.kill(pid, 0)passed after the caller's turn was over. Retrieval:agentdescent --store "$STORE" status fake-run-1 --json→"state": "running", "round": 2right after the handoff,"state": "finished", "round": 3after it ended, andshow fake-run-1 --json --no-diffresolves the same id.waitingwith a run id and round; no worker slot is keptwaiting.jsoncarriesrun_id: fake-run-1,round: 2,run_dir,driver,session,log; the gated smoke asserts those fields andelapsed < 2.0(measured 0.06 s), so the turn is over before the run starts its second round.dsh/agentdescentprocess after the Ticket ends or the run completeskill -KILL -$wrapper_pidon the worker's whole process group mid-run → wrapper dead, run alive (that is the handoff, tracked bywaiting.json), session alive; after completion → run pid not alive,pgrep -f "fake_run.py $STORE"= 0,tmux has-session= none. Repeated with tmux offPATH(thesetsiddriver): same survival under the group kill, no process and no session after completion. Not covered: a run killed by OOM or a crash leaves a dead pid instatus.jsonuntilresume— the restart row the page documents.pytest -o addopts="" -q tests/test_dsh_superagent_run.py→ 1 passed, 1 skipped;AGENTDESCENT_LONG_RUN_SMOKE=1on the same file → 2 passed. Nothing in the path opens a socket, starts a daemon or schedules a tick; the store directory is the handoff.The two fixes in this tip
f8db0e3c—tmux has-session -t ad-runsis a prefix target, so with anunrelated
ad-runs-zzzon the server the check returned 0 and the smoke couldreport, or wait on, a session the wrapper never started. That is a real
defect, not test jitter: the target is now
=ad-runs, which asks for exactlythat name.
b05c3f9d— the assertion was a single instant:status.jsonsaysfinishedbefore the run's process exits and tmux reaps the session a moment later, so
the check read a live run as an orphan. Rates at
43155caaas written: 3 of 12replicas failed; with a 1 s sleep before the check, 1 of 12 still failed (so a
sleep is not the fix, and one sample — 2 passed — was never a rate). Gated
pytest failed 2 of 4 at load 123 and passed 8 of 8 at load 17–30. The check is
now a bounded poll (15 s, 100 ms steps); its failure text carries the session
name, the wait and the last
has-sessionreading. A fake that holds itssession for 60 s still fails at the deadline, so the check is not weakened.
and the residual "no orphan" claim rests on the pid probe, not on the session.
The review fixes in this tip
976dc112— the three rows that pin the findings, red atb05c3f9d.d4fa76dc— the page and the wrapper's own usage now name the run's--storeas well as the report's, so
evolvewrites into the handoff's store insteadof the default one; the wrapper snapshots the store before launching and
accepts only a
status.jsonthat appears or changes after that, so a reusedstore cannot lend the report an earlier run; and the tmux session name carries
a per-launch suffix, so two handoffs into stores both called
runsget twosessions.
The fix at
c69767d7--budget 5with--usd-per-call 0.01.runstore.require_budget_pairrefuses a dollar budget with no per-call price,so the documented
evolve spec.json --detach --budget 5exited 2 beforecreating a run: the command the section tells a worker to copy could not start
one, and the "run stops itself and writes why" row rested on a flag the CLI
refused.
tests/test_dsh_superagent_run.pyno longer asserts a substring. It parsesthe fenced block, hands the child's argv to the CLI's own parser and requires
the documented pair to satisfy
require_budget_pair, so a page that drops aflag the CLI needs fails the test instead of shipping. Red at
d4fa76dc,only the page unchanged:
1 failed, 1 passed, 3 skipped,AssertionError: the page passes --budget 5.0 without --usd-per-call, so the documented command exits 2 and starts nothing.into one store inside the wait window both read that store's
waiting.jsonand the second reported the first's run. A documented expectation, not a code
change in this PR.
What this PR does not close on
#964: a child that dies immediately stillleaves the wrapper exiting 0 with
waiting, and the host-restart and OOM casesthe page documents are not exercised here, so the wrapper's own fail-closed
handling for a dead child is not part of this change. A reused store no longer
lends the report an earlier run --
d4fa76dccloses that. The four acceptancerows above are met; the remaining gap is why the issue stays open rather than
being closed by this PR.
The committed test pins the wrapper/page shape, the report fields, the
non-blocking return and the tmux-session check. The
status/showread and theprocess-group-kill probe above are commands measured at this tip, not assertions
in the test file.
The two commands behind rows (a) and (c):
Red at main
0c9e936d(test file only)Fresh tree at
0c9e936d, onlytests/test_dsh_superagent_run.pycopied in:The first is
AssertionError: no …/scripts/dsh-superagent-run.sh; the second isbash: …/scripts/dsh-superagent-run.sh: No such file or directory(exit 127).Red for the review's three rows, at
b05c3f9dOnly the three rows added to
tests/test_dsh_superagent_run.py:The reused-store row reproduces the review's repro at
b05c3f9d--found: truefor a run that had finished before the handoff started:
The Lead re-ran the same file copied onto a clean
b05c3f9dtree, independently(tmux 3.2a): 3 failed, 2 passed, with
duplicate session: ad-runs/tmux refused the session ad-runson the session row.Verification at the tip
Tip SHA: c69767d
Ran:
pytest -o addopts="" -q tests/test_dsh_superagent_run.pyat this tip →2 passed, 3 skipped (the three smoke rows are gated)
AGENTDESCENT_LONG_RUN_SMOKE=1 pytest -o addopts="" -q tests/test_dsh_superagent_run.pyat this tip → 5 passedpytest -o addopts="" -q tests/test_dsh_superagent_run.py tests/test_docs_links.py tests/test_site_navigation.py tests/test_changelog_fragments.py tests/test_repo_hygiene.pyat this tip → 73 passed, 3 skippedpython -m tools.gen_api_docs --check→docs/api.md is up to date;python -m tools.changelog_fragments check→34 changelog fragment(s) OKmkdocs build --strictat this tip → exit 0,Documentation built in 17.83 seconds(the same venv as the tests, with the material font cache in place)the documented command, end to end and unmodified from the page except for
the spec it names, through the wrapper, on this host:
The wrapper returned in 0.5 s with
found: trueand the run's own id; thespec is
agentdescent.demo.build's offline one (agentdescent.demo:offline_agent,rounds=3), so nothing here calls a model, and$0.10iscalls × --usd-per-call, which is the pair the page now documents.the new assertion is red where the report said the defect was: the test file
copied onto a clean
d4fa76dctree with the page unchanged →1 failed, 1 passed, 3 skipped, theAssertionErrorquoted above.Not run:
pytest -q— the docker,test_era_*,test_genesis_example,wheel-smoke and read-only-
$HOMElegs are environmental on this host; CI'stest (3.9)–(3.13)is the measured line§6, not exercised
the test file; the committed test pins the handoff command's flags, not the
status/showlinesmeasured, but no test races two launches
Size: 4 files,
+451/−0against base0c9e936d. Paths are T2 byscripts/**,but the size exceeds the T2 cap of 250 lines, so the gate tiers it
risk:T3(the label on the PR).
Authorship
Every commit on this branch carries the repository account as its author:
HugoXK <ke.xu@zju.edu.cn>, orDay Dreamer <87640074+HugoXK@users.noreply.github.com>, the second address that also mapsto it. The repository's attribution is the GitHub author, and this branch is one
shared account, so the body names no seats -- a per-commit seat list is a claim
the commits cannot support, and two of its entries were disputed.
Claim-Id: #964