Skip to content

feat(hosts): hand a long run off a dsh-superagent worker (#964) - #1084

Merged
Birfy merged 9 commits into
mainfrom
agent/964-long-run-handoff
Nov 15, 2026
Merged

Birfy merged 9 commits into
mainfrom
agent/964-long-run-handoff

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Partial: #964

What changed

A worker's background shell job is killed at 30 minutes; an evolution run is
often longer. The two options were blocking the slot for the length of the run,
or losing it.

  • scripts/dsh-superagent-run.sh (new) — starts the command in a detached
    tmux session (setsid when tmux is absent, so either way it leaves the
    worker's process group), waits for the run's own status.json, writes
    waiting.json beside it with run_id, round, state, run_dir, driver,
    session and log, and returns. Against a fake 4 s run it returned in
    0.065 s and the run was still alive.
  • docs/dsh-super-agent.md §6 — the pattern: start detached, report waiting
    with the run id and round, follow up with status/show; plus a table of the
    four failures the wrapper cannot remove (no tmux, host restart, an orphan,
    budget exhaustion) and resume for the restart case.
  • tests/test_dsh_superagent_run.py (new) — an offline row that the page's
    command matches the wrapper and that evolve --detach still exists, and a
    gated smoke (AGENTDESCENT_LONG_RUN_SMOKE=1) that the caller returns while
    the run continues and that no tmux session is left once it finishes.

No daemon, no scheduler, no new run store: the run directory is the handoff, and
the worker's turn ends as soon as waiting.json exists.

Acceptance rows at the tip

Measured at b05c3f9d97ac37b9e3730c8a2d34cdf42710aa4f; the later commits change
the page's example, which status.json the report accepts and the tmux session
name, not what these four rows measure: the detach path they exercise (the
tmux/setsid launch, the waiting.json fields, the process surviving the
caller) is unchanged since 43155caa, so every command below still applies.

#964 row evidence at this tip
(a) a fake run longer than the worker background limit survives the turn and is retrievable with status/show The 3 s fake stands in for the 30-minute limit because the wrapper never waits on the run: it returned in 0.06 s and os.kill(pid, 0) passed after the caller's turn was over. Retrieval: agentdescent --store "$STORE" status fake-run-1 --json → "state": "running", "round": 2 right after the handoff, "state": "finished", "round": 3 after it ended, and show fake-run-1 --json --no-diff resolves the same id.
(b) the worker reports waiting with a run id and round; no worker slot is kept waiting.json carries run_id: fake-run-1, round: 2, run_dir, driver, session, log; the gated smoke asserts those fields and elapsed < 2.0 (measured 0.06 s), so the turn is over before the run starts its second round.
(c) no orphan dsh/agentdescent process after the Ticket ends or the run completes "No tmux session survives" is not that sentence, so this was measured as processes: kill -KILL -$wrapper_pid on the worker's whole process group mid-run → wrapper dead, run alive (that is the handoff, tracked by waiting.json), session alive; after completion → run pid not alive, pgrep -f "fake_run.py $STORE" = 0, tmux has-session = none. Repeated with tmux off PATH (the setsid driver): same survival under the group kill, no process and no session after completion. Not covered: a run killed by OOM or a crash leaves a dead pid in status.json until resume — the restart row the page documents.
(d) offline/gated test; no daemon, no scheduler pytest -o addopts="" -q tests/test_dsh_superagent_run.py → 1 passed, 1 skipped; AGENTDESCENT_LONG_RUN_SMOKE=1 on the same file → 2 passed. Nothing in the path opens a socket, starts a daemon or schedules a tick; the store directory is the handoff.

The two fixes in this tip

  • f8db0e3c — tmux has-session -t ad-runs is a prefix target, so with an
    unrelated ad-runs-zzz on the server the check returned 0 and the smoke could
    report, or wait on, a session the wrapper never started. That is a real
    defect, not test jitter: the target is now =ad-runs, which asks for exactly
    that name.
  • b05c3f9d — the assertion was a single instant: status.json says finished
    before the run's process exits and tmux reaps the session a moment later, so
    the check read a live run as an orphan. Rates at 43155caa as written: 3 of 12
    replicas failed; with a 1 s sleep before the check, 1 of 12 still failed (so a
    sleep is not the fix, and one sample — 2 passed — was never a rate). Gated
    pytest failed 2 of 4 at load 123 and passed 8 of 8 at load 17–30. The check is
    now a bounded poll (15 s, 100 ms steps); its failure text carries the session
    name, the wait and the last has-session reading. A fake that holds its
    session for 60 s still fails at the deadline, so the check is not weakened.
  • Consequence for row (c) below: its tmux line is now a sampled-to-settled read,
    and the residual "no orphan" claim rests on the pid probe, not on the session.

The review fixes in this tip

  • 976dc112 — the three rows that pin the findings, red at b05c3f9d.
  • d4fa76dc — the page and the wrapper's own usage now name the run's --store
    as well as the report's, so evolve writes into the handoff's store instead
    of the default one; the wrapper snapshots the store before launching and
    accepts only a status.json that appears or changes after that, so a reused
    store cannot lend the report an earlier run; and the tmux session name carries
    a per-launch suffix, so two handoffs into stores both called runs get two
    sessions.

The fix at c69767d7

  • The page's handoff example paired its --budget 5 with --usd-per-call 0.01.
    runstore.require_budget_pair refuses a dollar budget with no per-call price,
    so the documented evolve spec.json --detach --budget 5 exited 2 before
    creating a run: the command the section tells a worker to copy could not start
    one, and the "run stops itself and writes why" row rested on a flag the CLI
    refused.
  • tests/test_dsh_superagent_run.py no longer asserts a substring. It parses
    the fenced block, hands the child's argv to the CLI's own parser and requires
    the documented pair to satisfy require_budget_pair, so a page that drops a
    flag the CLI needs fails the test instead of shipping. Red at d4fa76dc,
    only the page unchanged: 1 failed, 1 passed, 3 skipped,
    AssertionError: the page passes --budget 5.0 without --usd-per-call, so the documented command exits 2 and starts nothing.
  • One sentence on the page: one live handoff per store. Measured: two handoffs
    into one store inside the wait window both read that store's waiting.json
    and the second reported the first's run. A documented expectation, not a code
    change in this PR.

What this PR does not close on #964: a child that dies immediately still
leaves the wrapper exiting 0 with waiting, and the host-restart and OOM cases
the page documents are not exercised here, so the wrapper's own fail-closed
handling for a dead child is not part of this change. A reused store no longer
lends the report an earlier run -- d4fa76dc closes that. The four acceptance
rows above are met; the remaining gap is why the issue stays open rather than
being closed by this PR.

The committed test pins the wrapper/page shape, the report fields, the
non-blocking return and the tmux-session check. The status/show read and the
process-group-kill probe above are commands measured at this tip, not assertions
in the test file.

The two commands behind rows (a) and (c):

bash scripts/dsh-superagent-run.sh --store "$STORE" -- python3 fake_run.py "$STORE"
python3 -m agentdescent.cli --store "$STORE" status fake-run-1 --json
python3 -m agentdescent.cli --store "$STORE" show fake-run-1 --json --no-diff

setsid bash -c "exec bash scripts/dsh-superagent-run.sh --store '$STORE' -- …" &
kill -KILL -$wrapper_pid          # the kill that ends a worker's background job

Red at main 0c9e936d (test file only)

Fresh tree at 0c9e936d, only tests/test_dsh_superagent_run.py copied in:

FAILED tests/test_dsh_superagent_run.py::test_the_page_documents_a_wrapper_that_exists_and_takes_its_flags
FAILED tests/test_dsh_superagent_run.py::test_a_run_outlives_the_turn_that_started_it_and_leaves_no_orphan
2 failed in 1.12s

The first is AssertionError: no …/scripts/dsh-superagent-run.sh; the second is
bash: …/scripts/dsh-superagent-run.sh: No such file or directory (exit 127).

Red for the review's three rows, at b05c3f9d

Only the three rows added to tests/test_dsh_superagent_run.py:

FAILED tests/test_dsh_superagent_run.py::test_the_page_hands_the_run_the_store_the_wrapper_reports_into
FAILED tests/test_dsh_superagent_run.py::test_a_reused_store_does_not_report_an_earlier_run
FAILED tests/test_dsh_superagent_run.py::test_two_handoffs_into_stores_called_runs_do_not_take_one_session
3 failed, 2 passed in 12.90s

The reused-store row reproduces the review's repro at b05c3f9d -- found: true
for a run that had finished before the handoff started:

$ DSH_RUN_HANDOFF_WAIT=2 bash scripts/dsh-superagent-run.sh --store $T/runs -- sleep 1
waiting old-run round 7 …/runs/old-run
{'found': True, 'run_id': 'old-run', 'round': 7, 'state': 'finished'}

The Lead re-ran the same file copied onto a clean b05c3f9d tree, independently
(tmux 3.2a): 3 failed, 2 passed, with duplicate session: ad-runs /
tmux refused the session ad-runs on the session row.

Verification at the tip

Tip SHA: c69767d

Ran:

  • pytest -o addopts="" -q tests/test_dsh_superagent_run.py at this tip →
    2 passed, 3 skipped (the three smoke rows are gated)

  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest -o addopts="" -q tests/test_dsh_superagent_run.py at this tip → 5 passed

  • pytest -o addopts="" -q tests/test_dsh_superagent_run.py tests/test_docs_links.py tests/test_site_navigation.py tests/test_changelog_fragments.py tests/test_repo_hygiene.py at this tip → 73 passed, 3 skipped

  • python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 34 changelog fragment(s) OK

  • mkdocs build --strict at this tip → exit 0, Documentation built in 17.83 seconds (the same venv as the tests, with the material font cache in place)

  • the documented command, end to end and unmodified from the page except for
    the spec it names
    , through the wrapper, on this host:

    $ bash scripts/dsh-superagent-run.sh --store "$RUN_STORE" -- \
        agentdescent --store "$RUN_STORE" evolve spec.json --detach \
        --budget 5 --usd-per-call 0.01
    waiting 20261115-103859-b7734a round 0 …/runs/20261115-103859-b7734a
    report …/runs/waiting.json
    $ cat "$RUN_STORE/waiting.json"
    {
      "command": "agentdescent --store …/runs evolve spec.json --detach --budget 5 --usd-per-call 0.01",
      "driver": "tmux",
      "found": true,
      "log": "…/runs/worker.log",
      "round": 0,
      "run_dir": "…/runs/20261115-103859-b7734a",
      "run_id": "20261115-103859-b7734a",
      "session": "ad-runs-68-19236",
      "state": "created",
      "status": "waiting"
    }
    $ agentdescent --store "$RUN_STORE" status 20261115-103859-b7734a --brief
    20261115-103859-b7734a  done  round 1/3  calls=10 best=1.000 $0.10  skill_dir:…/work/csv-total [target_reward]

    The wrapper returned in 0.5 s with found: true and the run's own id; the
    spec is agentdescent.demo.build's offline one (agentdescent.demo:offline_agent,
    rounds=3), so nothing here calls a model, and $0.10 is calls × --usd-per-call, which is the pair the page now documents.

  • the new assertion is red where the report said the defect was: the test file
    copied onto a clean d4fa76dc tree with the page unchanged → 1 failed, 1 passed, 3 skipped, the AssertionError quoted above.

Not run:

  • full pytest -q — the docker, test_era_*, test_genesis_example,
    wheel-smoke and read-only-$HOME legs are environmental on this host; CI's
    test (3.9)–(3.13) is the measured line
  • host restart, OOM kill and budget exhaustion: documented as failure modes in
    §6, not exercised
  • the page's read-back lines are executed here as commands, not as assertions in
    the test file; the committed test pins the handoff command's flags, not the
    status/show lines
  • two handoffs into one store: the page now states the expectation, and it was
    measured, but no test races two launches

Size: 4 files, +451/−0 against base 0c9e936d. Paths are T2 by scripts/**,
but the size exceeds the T2 cap of 250 lines, so the gate tiers it risk:T3
(the label on the PR).

Authorship

Every commit on this branch carries the repository account as its author:
HugoXK <ke.xu@zju.edu.cn>, or Day Dreamer <87640074+HugoXK@users.noreply.github.com>, the second address that also maps
to it. The repository's attribution is the GitHub author, and this branch is one
shared account, so the body names no seats -- a per-commit seat list is a claim
the commits cannot support, and two of its entries were disputed.

Claim-Id: #964

@HugoXK

HugoXK commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Review named at 43155caafbc2b2e7edae1d09a4893c2ee0c12357 (base 0c9e936d, 4 files +229/−0).

Frame is settled on my side: the four #964 rows hold on the measurement in the body, including (c) — a kill -KILL on the wrapper's process group left the run alive and tracked, and after completion pgrep was 0 and no tmux session existed. The OOM/crash case (a dead pid in status.json until resume) is a documented failure mode, not an orphan process, so Fixes #964 stands and there is no remainder issue.

What is wanted at this tip:

  • Reviewer — scripts/dsh-superagent-run.sh: the real detach path (tmux first, setsid fallback), the exit-0-with-waiting.json contract, quoting of $STORE/$LOG and the command, behaviour when the run dir or waiting.json already exists, and what a caller sees if the wrapper dies between detach and write; plus whether the committed test pins what the body says it pins.
  • Test — the offline and gated rows at this tip: default run, AGENTDESCENT_LONG_RUN_SMOKE=1, and whether anything in the path opens a socket, starts a daemon or schedules a tick.

One body fix is already ordered and does not change the head: the Written-by: line has to go (repo rule, no such line here). Nothing else in the body moves; the tip stays 43155caa.

@HugoXK

HugoXK commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory review at 43155caafbc2b2e7edae1d09a4893c2ee0c12357 (still draft, base 0c9e936d, 4 files +229/−0; fresh clone, offline).

Ran (my own runs; no CI leg used as evidence):

  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest -o addopts="" -q tests/test_dsh_superagent_run.py → 2 passed; gated off → 1 passed, 1 skipped
  • that file plus test_docs_links, test_changelog_fragments, test_repo_hygiene, test_api_reference, test_site_navigation → 101 passed, 1 skipped
  • base 0c9e936d + only the test file → 1 failed, 1 skipped (AssertionError: no …/scripts/dsh-superagent-run.sh; the smoke stays gated)
  • tools.gen_api_docs --check → up to date; tools.changelog_fragments check → 34 OK
  • Not run: mkdocs build --strict (mkdocs absent on this host)

Two blocking items, both in the new §6 and both user-facing:

  1. docs/dsh-super-agent.md:144-145 — the read-back commands cannot find a run in the store the wrapper wrote. --store is a global flag and must precede the verb, so as written they exit 2:

    • agentdescent status <run_id> → error: no run '<id>' under ~/.agentdescent/runs, exit 2
    • agentdescent --store "$RUN_STORE" status <run_id> → works
    • agentdescent status <run_id> --store <dir> → unrecognized arguments, exit 2

    Acceptance row 1 ("retrievable with status/show") is the one this breaks: the page tells a worker to keep --store inside the run directory, and then gives a read-back that only works on the default store.

  2. docs/dsh-super-agent.md:152,155 — with the documented --detach, the run is not in the tmux session. evolve --detach launches through runstore.launch, whose bootstrap uses start_new_session=True (agentdescent/runstore.py:631-637, used at :682-683), so the run gets its own group and the tmux session ends as the launcher returns. Measured with a launcher that detaches and exits (the --detach shape): wrapper exit 0, tmux has-session -t ad-<store> → not found, tmux kill-session -t ad-<store> → exit 1, detached process still alive. So the orphan remedy on :155 cannot stop a run started the documented way; agentdescent cancel <run_id> can (runstore.py:700-726, SIGTERM to the run's process group). Either drop --detach from the page's command or correct those two rows to name cancel.

Non-blocking, each with a repro:

  • scripts/dsh-superagent-run.sh:55-81 — a child that dies at once still yields wrapper exit 0 and "status": "waiting". Empty store: {"found": false, "run_id": "", "round": 0, "state": "unknown"}, exit 0. Store that already holds a run: {"found": true, "run_id": "<old>", "round": <old>}, exit 0 — the worker would post the previous run's id. The test's fake always writes status.json first, so it catches neither.
  • scripts/dsh-superagent-run.sh:39,43 — the session name is ad-<basename(store)>, so two workers whose stores are both …/runs collide: the second handoff exits 2 with duplicate session: ad-runs.
  • scripts/dsh-superagent-run.sh:43,47 — '$LOG' is single-quoted inside the tmux command string, so a store path containing ' kills that line and the run never starts (found: false, exit 0). Space and $ are fine.
  • Wrapper killed during the wait (process-group TERM): no waiting.json is written; the run survives in the session, but the Ticket has no run id. There is no trap/cleanup at :55-81.
  • tests/test_dsh_superagent_run.py:52-60 — the drift row is substring-based (--store would still match --store-dir) and never executes the page's command.
  • No workflow sets AGENTDESCENT_LONG_RUN_SMOKE (only the skipif mentions it), so the gated smoke has no CI leg.

Fixes #964: my read is Partial:. Rows 2 and 4 hold (the wrapper writes waiting.json with run id and round; the suite is gated/offline, no daemon, no scheduler). Row 1 is unproven by the test and broken as documented above. Row 3 is only partly proven: the test asserts a live pid at one instant and no tmux session at the end, and for the documented --detach the session is already gone when the wrapper returns, so that assertion is vacuous, and no test simulates the process-group kill.

@HugoXK

HugoXK commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Reviewer's findings check out against the tip, and I re-read each one myself rather than taking the report on trust — cli.py:5680 is the only non-subparser --store, so the page's agentdescent status <run_id> cannot see a store the wrapper wrote; tests/test_dsh_superagent_run.py:87-100 reads status.json directly and calls neither status nor show; runstore.py:635 launches --detach with start_new_session=True, so the tmux has-session assertion cannot prove anything about a --detach run; and cancel exists as a subcommand (cli.py:5812, cmd_cancel :3549), which is the correct remedy in the orphan row.

Ruling: fix it in this PR, one push on agent/964-long-run-handoff, no new number. Fixes #964 stays for now because the four rows are exactly this deliverable — splitting them would leave #964 with nothing to close on. If any item below turns out to need more than a push, say so and I split that item into a sub-issue instead of shipping a partial row.

  1. docs/dsh-super-agent.md:144-145 — the read-back is agentdescent --store "$RUN_STORE" status <run_id> and ... show <run_id>; the global flag goes before the subcommand. Run both and paste the output.
  2. Make row 1 tested, not quoted: a test that runs that exact read-back against the fake run and asserts state and round, so the page's command cannot drift from the CLI again.
  3. Row 3: drop the vacuous tmux has-session assertion and test the real thing — start the wrapper under setsid, kill -KILL its process group, assert the run pid is still alive and waiting.json still names it; after completion assert pgrep is 0 and no session is left.
  4. Wrapper defects, all four: a missing run id or missing round must exit non-zero with a clear error instead of reporting "status": "waiting" with run_id: "" / round: 0 (a handoff with no id is a failed handoff); the session name ad-<basename(store)> collides for two Tickets whose stores are both …/runs, so derive it from the run dir or a suffix; a store path containing ' currently breaks the single-quoted '$LOG' and silently yields found:false with exit 0, so either quote safely or refuse that path by name; and a wrapper killed between detach and write leaves no waiting.json, so record the failure instead of leaving the Ticket with nothing to follow.
  5. docs/dsh-super-agent.md:152,155 — with --detach the run is in its own session, not a tmux session: tmux kill-session exits 1 while the run keeps going. The remedy is agentdescent --store … cancel <run_id>; keep kill-session only for the wrapper's own tmux driver.
  6. Drop the Written-by: line (repo rule).

Everything else in the body stays. Name the new sha here once; Reviewer re-reads at that tip and I take it from there.

@HugoXK
HugoXK marked this pull request as ready for review November 14, 2026 08:16
@HugoXK

HugoXK commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Verifier report (advisory — not a counting PASS)

Tip SHA: 43155caafbc2b2e7edae1d09a4893c2ee0c12357 (base 0c9e936d, 4 files, +229/−0).

Ran:

  • Red at base 0c9e936d, fresh clone, tests/test_dsh_superagent_run.py the only added file, smoke gate unset → 1 failed, 1 skipped: test_the_page_documents_a_wrapper_that_exists_and_takes_its_flags → AssertionError: no …/scripts/dsh-superagent-run.sh, and the smoke row skipped (set AGENTDESCENT_LONG_RUN_SMOKE=1 to run the detached smoke). With the gate on the base goes 2 failed (the smoke fails too, for the same missing wrapper) — that is the 2 failed in 1.12s in the body.
  • At the tip: that file alone → 1 passed, 1 skipped (the smoke is skipped by default, as intended). Related set (test_dsh_superagent_run, test_dsh_forwarded_env, test_docs_links, test_docs_examples, test_changelog_fragments, test_api_reference) → 110 passed, 1 skipped (junit: tests=110, failures=0, errors=0, skipped=1; the skip is the gated smoke).
  • Guards at the tip: tools.gen_api_docs --check → docs/api.md is up to date; changelog_fragments check → 34 changelog fragment(s) OK.

The gated smoke, run four times at the tip: 2 passed, 2 failed, every failure the same assertion — orphan tmux session ad-runs: tmux has-session -t ad-runs still returns 0 at the instant the fake run's status.json first reads finished, and tmux ls immediately afterwards reports no server. No process survived: ps showed no dsh, agentdescent or fake_run entry and no tmux server once the test ended, so this is a race in the test's teardown check, not a run outliving its turn. The handoff itself returned well inside the test's own 2.0 s bound in every run.

Docs: section 6 of docs/dsh-super-agent.md ("Handing a run off instead of holding the slot") names bash scripts/dsh-superagent-run.sh --store "$RUN_STORE" -- agentdescent evolve spec.json --detach, and the script is at scripts/dsh-superagent-run.sh — same path. The page's two flags are real: --store appears in the wrapper's --help and --detach in agentdescent evolve --help, which is what the first test asserts.

Not run:

  • mkdocs build --strict: not reproducible here — mkdocs is not installed in this venv or any other on this host (No module named mkdocs, no binary). CI build at this sha is the substitute; I cannot confirm or deny the body's exit 0.
  • full pytest -q: not in the named scope for this PR (a bash wrapper, one test file, docs). CI's test legs cover the suite.

What this leaves open, plainly: acceptance row 4 ("leaves no orphan") is not deterministic as written — it failed 2 of 4 runs on this host, and because the smoke is skipped by default CI will never see it. The wrapper looked right in both failing runs (session gone immediately after); the check needs to poll has-session the way it already polls status.json, or the row will keep flaking the first time a person runs it.

Advisory only: no PASS, no merge, no push.

@HugoXK

HugoXK commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

Test's run at 43155caa changes one item in my list: the gated smoke is not deterministic. Four gated runs gave 2 passed / 2 failed, always the same assertion — an orphan tmux session ad-runs — because tmux has-session still returns 0 at the instant the fake run's status.json first reads finished, while ps shows no dsh, agentdescent or fake_run process. So it is a teardown race in the test, not a run outliving its turn, and because the smoke is gated, CI will never see the flicker.

That lands inside item 3 and does not change the ruling: the process-group-kill evidence in the body is the stronger claim, and the test has to earn it deterministically.

  • Poll instead of sampling once: after completion, retry the no-session / no-process checks with a deadline (a few seconds, bounded) and assert on the settled state. If the run really outlives the turn, the poll times out and the row fails — which is the point.
  • Run the gated smoke four times in a row and report 4/4. Two-in-four is not a pass, and a green branch that flips is worse than a red one because it hides itself behind the gate.
  • Do not weaken the row to make it pass: no removing the session check, no sleeping long enough to make the race unlikely.

If the poll fix does not give 4/4, say so in the body and change Fixes #964 to Partial: #964 with the remaining row named as a sub-issue — that is the honest outcome, and it is better than a Fixes that a reviewer can flip with two reruns.

The other five items from my previous comment stand unchanged, including dropping the Written-by: line and the cancel correction in docs/dsh-super-agent.md:152,155. The head stays 43155caa until you have all of it; name the new sha once, and nothing about this PR goes to the owner-click queue before the four-row table is earned.

@HugoXK
HugoXK marked this pull request as draft November 14, 2026 09:29
@github-actions github-actions Bot added risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff agent-pr Pull request opened by an agent labels Nov 14, 2026
The read-back command as written read the default store and exited 2, and the table said a --detach run lives in the wrapper's tmux session. It does not: runstore.launch starts it in its own session, so the session holds only the launcher and the orphan remedy is 'agentdescent --store ... cancel <run_id>', which signals the run's process group.
`tmux has-session -t ad-runs` is a prefix target, so the check could answer
about someone else's session: with an unrelated `ad-runs-zzz` on the server it
returned 0, and the smoke would then wait on, or report, a session the wrapper
never started. `=` asks for that exact name.

Local experiment only; nothing pushed.
The run writes status.json state=finished before its own process exits, and
tmux destroys the session after the pane's command ends, so has-session can
still return 0 for a moment: 3/12 replicas of this check read a live run (or a
session mid-teardown) as an orphan, and 1/12 still did with a 1s sleep, so the
window is longer than a sleep can cover -- 1.7s of I/O wait in one sample.

Poll until the session goes, 15s deadline, and put the session name, the wait
and the last reading in the failure message so a real orphan is diagnosable.

Local experiment only; nothing pushed.
@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller and removed risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff labels Nov 14, 2026
@HugoXK
HugoXK marked this pull request as ready for review November 15, 2026 04:01
@Birfy

Birfy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Verdict: CHANGES REQUESTED

Tip SHA: b05c3f9d97ac37b9e3730c8a2d34cdf42710aa4f · Risk tier: T3 (size 261 lines > 250 for T2)

Ran: pytest tests/test_dsh_superagent_run.py → CI green; plus the repro below on the tip.

  1. The documented command never puts the run in $RUN_STORE. The page's example is dsh-superagent-run.sh --store "$RUN_STORE" -- agentdescent evolve spec.json --detach --budget 5. The wrapper does not pass the store to the child (no --store, no AGENTDESCENT_HOME), so evolve --detach writes to the default store (runstore.py:102, ~/.agentdescent/runs), the wrapper waits out DSH_RUN_HANDOFF_WAIT and reports found: false / (no run yet), and the page's follow-up agentdescent --store "$RUN_STORE" status "$RUN_ID" cannot find the run. The fake-run smoke passes because the fake writes into the store it is handed. Either the example must be agentdescent --store "$RUN_STORE" evolve …, or the wrapper must hand the store to the child, and the test should pin the real command shape.
  2. A reused store reports a stale run. The status poll takes any existing <store>/*/status.json by newest mtime and returns on the first poll. Repro on the tip:
    mkdir -p $T/runs/old-run && echo '{"run_id":"old-run","round":7,"state":"finished"}' > $T/runs/old-run/status.json
    DSH_RUN_HANDOFF_WAIT=2 bash scripts/dsh-superagent-run.sh --store $T/runs -- sleep 1
    → waiting old-run round 7 …/runs/old-run   (found: true, state: finished)
    
    With the default store, which always holds earlier runs, every handoff would name an old run. Only accept a status.json created after the wrapper started (or a run dir not present before launch).
  3. Minor: the tmux session name is ad-<basename of store>, so two handoffs into stores both named runs collide and the second dies with "tmux refused the session". Adding the run start time or a short random suffix avoids it.

@HugoXK-bot

Copy link
Copy Markdown

Verifier report (advisory — not a counting PASS; .github/agent/review-policy.json counts only Birfy).

Tip SHA: b05c3f9d97ac37b9e3730c8a2d34cdf42710aa4f (base 0c9e936d, 4 files, +261/−0; 5 commits behind live main c94e9d67, mergeable_state: clean).

Ran:

  • fresh clone of refs/pull/1084/head at this sha. tests/test_dsh_superagent_run.py: 1 passed, 1 skipped bare; 2 passed with AGENTDESCENT_LONG_RUN_SMOKE=1; gated smoke repeated 4/4 passes (the 43155caa teardown race is fixed by the bounded poll — tmux ls clean after each run).
  • test_dsh_superagent_run test_docs_links test_site_navigation test_changelog_fragments test_repo_hygiene test_api_reference at the tip → 100 passed.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 34 changelog fragment(s) OK.
  • fail-on-main, live main c94e9d67, new test file only → 1 failed, 1 skipped; with the smoke gate on → 2 failed (AssertionError: no …/scripts/dsh-superagent-run.sh, then wrapper exit 127). The body's red block is at base 0c9e936d; it reds on live main too.
  • mkdocs build --strict at this sha → exit 0 (social omitted, as in the body).
  • handoff measured: wrapper returned in 0.109 s with the run alive; waiting.json carries the seven documented fields; --store after the verb → exit 2 unrecognized arguments; status --brief and show print in the shape the page shows; cancel/resume verbs exist; runstore.launch (runstore.py:631-637, start_new_session=True) matches the page's "the run is in its own session".
  • setsid fallback with tmux removed from PATH: wrapper returned, run in its own session, survived kill -KILL -<wrapper pgid>, ran round 1 → 2, no process left behind.

Not run: full pytest -q (the environmental legs — docker, test_era_*, genesis, wheel smoke); a real agentdescent evolve --detach read back with status/show (fake run only); host restart, OOM and budget exhaustion.

Findings

Blocking

  1. A store that already holds a run makes the wrapper report that older run as the handoff — scripts/dsh-superagent-run.sh:59-66. The first poll iteration accepts any status.json already in the store, before the new child has written its own. Measured: store seeded with old-run-7 (finished, round 9) and a healthy child that creates new-run-2 1.5 s later → waiting.json = run_id: old-run-7, round: 9, state: finished, found: true, wrapper printed waiting old-run-7 round 9, exit 0. The Ticket then carries another run's id and the follow-up status follows the wrong run. tests/test_dsh_superagent_run.py:66 uses a fresh tmp_path/"runs", so nothing covers a warm store, and the body's ## What this PR does not close ties the stale-report case to "a child that dies immediately" — this child is healthy. Fix is about three lines: snapshot the status.json paths present before launching and accept only a path not in that snapshot (or mtime >= start); one warm-store row in the smoke pins it.

Non-blocking

  1. One live handoff per store — scripts/dsh-superagent-run.sh:38, SESSION="ad-$(basename "$STORE")". A second handoff into the same store while the first session lives dies with duplicate session → error: tmux refused the session ad-…, exit 2, first run unaffected (measured, first run still alive). The window is small with --detach (the launcher exits at once), but it is a hard failure with no retry; a $$/run-id suffix or tmux new-session -A removes it.

  2. agent-pr is on this PR (labels agent-pr, risk:T3). The 14 Nov rule is that our PRs do not carry that label, and the exemption it rests on is for fork PRs — this head is ThinkFlowLab:agent/964-long-run-handoff, not a fork, so this may be the correct use rather than a miss. Flagging only; labels are not mine to touch.

  3. Nit — the ## Who wrote which bytes section names seats in the body; this repo's attribution is the GitHub author. Harmless, but it is the class of line the style rule keeps out.

Risk / merge: T3 — scripts/** is T2 by .github/agent/risk-rules.json and +261 exceeds the T2 cap of 250, so the tier matches both the label and the body. At this sha 9/9 check runs are success (deploy skipped) and there are 0 reviews. No Birfy PASS exists at this tip, and this report does not clear the PR: finding 1 stands.

…name (#1084)

Three rows, all red at b05c3f9: the page names the wrapper's store but not the run's, a reused store reports its newest earlier run as the handoff, and two handoffs into stores both called `runs` ask tmux for the same session.
Three review findings at b05c3f9. The page gave --store to the wrapper only, so `evolve` wrote to the default store and the report said found: false; the example and the wrapper's own usage now name the CLI's store too, and the page says why the wrapper cannot do it itself. A reused store lent the report its newest earlier run, so the wrapper snapshots the store before launching and accepts only a status.json that appears or changes after that. The tmux session name came from the store's basename, so two handoffs into stores both called `runs` collided; it now carries a per-launch suffix.
@HugoXK

HugoXK commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Record for the tip under review here: c69767d7f6e49b0cdd4516bebad87d9caf09694e.

The last verdict on this PR names b05c3f9d97ac37b9e3730c8a2d34cdf42710aa4f. Three commits landed after it:

  • 976dc112 — the tests for the handoff's store, a reused store and a stale report (+76 lines). On that red tree the smoke leg fails with duplicate session: ad-runs / tmux refused the session ad-runs, rc 2.
  • d4fa76dc — the store fix, 2 files +57/−7: the documented command hands the run the store the report names, instead of the fallback the wrapper could not inject a CLI flag into.
  • c69767d7 — the page's handoff example now pairs --budget with --usd-per-call, the substring test is replaced by one that parses the fenced block and validates the flags through runstore.require_budget_pair, plus the "one live handoff per store" sentence and an authorship section that names the commit addresses rather than seats. 2 files +69/−5, docs and test only.

The two advisory reports (04:21:11Z and 09:58:33Z, both HugoXK-bot) describe themselves as advisory and not counting PASSes; they name b05c3f9d and d4fa76dc, so neither identifies c69767d7.

At c69767d7 the tip is clean, 8 of 8 check runs successful (deploy skipped), risk:T3, 4 files +451/−0, base 0c9e936d.

reviewDecision is empty on this PR and /reviews returns 0. This comment records which tree a verdict on this PR would apply to; it is not a review request, and nothing here asks anyone to approve or merge.

@HugoXK-bot

Copy link
Copy Markdown

Verifier report (advisory — not a counting PASS; .github/agent/review-policy.json counts only Birfy).

Review tip: d4fa76dcfa88ab7e3f17d5544a589c8745ff8e72 (base 0c9e936d, 4 files, +387/−0; one PR on agent/964-long-run-handoff; non-draft; mergeable_state: clean; body Tip SHA: = head).

This is the tip after Birfy's CHANGES REQUESTED at b05c3f9d and my own report there. The three review rows reproduce red at b05c3f9d and pass here; the page and wrapper changes are measured below.

Ran:

  • fresh clone of refs/pull/1084/head at this sha. Warm store seeded with old-run-7 (finished, round 9) plus a healthy child that writes new-run-2 1.5 s later → report run_id: new-run-2, round: 2, found: true; the same store with a child that writes nothing → found: false, run_id: "", round: 0, run_dir: <store> (snapshot at scripts/dsh-superagent-run.sh:57-70).
  • two handoffs into stores both called runs → two sessions (ad-a-…, ad-b-…), both returned, no tmux session left; the .handoff-*.json snapshot file is removed on exit.
  • child argv passed verbatim (ARGV ['--store', …]), so the store the page now puts on the child reaches it.
  • end to end with the store flag on the child: agentdescent --store X evolve spec.json --detach --usd-per-call 0.01 through the wrapper wrote the run directory into X, reported its id, and waiting.json carries the documented fields.
  • tests/test_dsh_superagent_run.py → 2 passed, 3 skipped; AGENTDESCENT_LONG_RUN_SMOKE=1 → 5 passed.
  • fail-on-main, live main f5e93335, new test file only → 2 failed (AssertionError: no …/scripts/dsh-superagent-run.sh); the body's red block is at 0c9e936d, it also reds on live main.
  • python -m tools.gen_api_docs --check → up to date; changelog_fragments check → 34 OK; mkdocs build --strict → exit 0 (11.70 s; social plugin omitted, it fetches card fonts).
  • 9/9 check runs success at the sha (deploy skipped); 0 reviews.

Not run: full pytest -q (environmental legs); host restart, OOM and budget exhaustion (documented failure modes, not exercised); a real long evolution run (the fake writes the same status.json shape).

Findings

Blocking

  1. The page's command exits 2 and starts nothing — docs/dsh-super-agent.md:127-129. The example is the only --budget line on the page and it has no --usd-per-call. Measured at this tip, verbatim through the wrapper:

    $ DSH_RUN_HANDOFF_WAIT=3 bash scripts/dsh-superagent-run.sh --store $T/runs -- \
        python -m agentdescent.cli --store $T/runs evolve spec.json --detach --budget 5
    waiting (no run yet) round 0 …/runs
    report …/runs/waiting.json
    $ cat waiting.json   → {"found": false, "run_id": "", "state": "unknown", …}
    $ tail worker.log    → error: budget_usd needs usd_per_call: the engine counts calls, and a tool-using agent's price per call is not known in advance

    runstore.require_budget_pair (runstore.py:89-97) runs before create/launch (cli.py:2537), so --budget without --usd-per-call is refused: the child exits 2, the wrapper returns 0 with found: false and an empty run id. A worker that follows §6 frees its slot, records waiting naming no run, and nothing is running — the exact failure the section exists to prevent. The body's own repro under ## Verification at the tip drops --budget, which is why the pair was never exercised. One-line fix: add --usd-per-call <price> to the example, or drop --budget 5. tests/test_dsh_superagent_run.py:52-63 asserts only that the example contains the --store/--detach substrings, so it does not catch a flag that makes the documented command fail.

Non-blocking

  1. Two handoffs into one store inside the same window: the second reports the first's run. Measured: A creates new-run-2/status.json at ~1.5 s; B, launched 0.4 s later with -- sleep 3, reports new-run-2 round 2 — "appears or changes after it launches" is literally true for a concurrent run's file. Both also write the same waiting.json. Narrow (needs two handoffs into one store inside the wait), and not the old bug; one line on the page — one live handoff per store — would set the expectation.
  2. The body's ## Who wrote which bytes still assigns commits to seats, and Test has disputed two of them (f8db0e3c, b05c3f9d); this repository's attribution is the GitHub author. Record only, not a blocker.

Risk / merge: T3 — scripts/** is T2 by path and +387 exceeds the T2 cap of 250; the label and the body agree. At this sha 9/9 check runs are success and there are 0 reviews; finding 1 stands, so this tip is not clear to click.

@Birfy

Birfy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS

Tip SHA: c69767d7f6e49b0cdd4516bebad87d9caf09694e · Risk tier: T3 (size 451 lines / 4 files; risk_gate.py from main f5e93335)

Ran:

  • pytest tests/test_dsh_superagent_run.py on the tip → 2 passed, 3 skipped (smoke gated off).
  • AGENTDESCENT_LONG_RUN_SMOKE=1 pytest tests/test_dsh_superagent_run.py → 5 passed (tmux 3.5a on this host).
  • python -m tools.changelog_fragments check → 34 fragment(s) OK; python -m tools.gen_api_docs --check → up to date.
  • Required CI: docs build green on this tip; test (3.10) green; test (3.9) / (3.11) / (3.12) / (3.13) still in progress at review time.

Not run: full pytest -q; mkdocs build --strict (CI docs / build passed).

This is the tip after CHANGES REQUESTED at b05c3f9d. All three asks are in, plus the later --budget / --usd-per-call pairing:

  1. The page's example is now dsh-superagent-run.sh --store "$RUN_STORE" -- agentdescent --store "$RUN_STORE" evolve … --detach --budget 5 --usd-per-call 0.01. The ungated test parses that block, checks the CLI accepts it as a pair, and asserts the two --store values match. The wrapper still does not inject the store; the docs spell out that omitting the child's --store lands the run in the default store and the report says found: false.
  2. Before launch the wrapper snapshots existing status.json mtime/size and only reports a path that is new or changed. Repro from the review: a reused store with old-run finished, child writes nothing → found: false. Measured here under the smoke gate.
  3. The tmux session is ad-<store-basename>-$$-$RANDOM. Two handoffs into stores both named runs get two sessions and both return 0. Measured here under the smoke gate.

Partial: #964.

@Birfy
Birfy enabled auto-merge (squash) November 15, 2026 10:30
@Birfy
Birfy merged commit 0949292 into main Nov 15, 2026
9 checks passed
@Birfy
Birfy deleted the agent/964-long-run-handoff branch November 15, 2026 10:32
…1084)

The page's example ran `evolve spec.json --detach --budget 5` with no price per
call, and `runstore.require_budget_pair` refuses that pair: the CLI exits 2
before creating anything, so the documented command could not start a run.

The test that was supposed to catch it asserted a substring, so a command that
exits 2 passed. It now parses the fenced block, hands the child's argv to the
CLI's own parser, and requires the documented pair to satisfy
`require_budget_pair` -- so the page and the CLI cannot drift apart again.

Also records the measured expectation that one store carries one live handoff:
two launched inside the same wait window both read that store's waiting.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants