fix(supervisor): treat a live heartbeat as liveness and keep watchdog stops resumable

[lishuoshuo-amd]

Summary

• Drive session-lock heartbeats from the same asyncio loop as the coordinator, so long awaited actions remain healthy while a blocked loop can still become WEDGED.
• Mark supervisor stop requests before SIGTERM and keep cooperative watchdog restarts non-terminal and resumable.
• Preserve legacy lock behavior, manual SIGTERM terminal handling, and terminal reports for coordinators that die without a supervisor stop request.

Tests

• python3 -m pytest -n 8 -q src/hyperloom/orchestrator/supervisor/tests/test_supervisor.py src/hyperloom/inference_optimizer/tests/test_session_lock.py src/hyperloom/inference_optimizer/tests/test_coordinator_runtime.py src/hyperloom/inference_optimizer/tests/test_run_optimize_exit_code_lock.py — 132 passed
• python3 -m ruff check on all changed files — passed
• Full non-E2E suite: 21,066 passed and 52 skipped; 18 unrelated environment/integration failures plus one missing hypothesis dependency prevented a clean full-suite result.

Scope

• Addresses supervisor-watchdog Findings 2 and 3 only. Finding 1 was fixed by [Refactor] Rework the enablement / bring-up round lifecycle #1409.
• No breaking API changes.

[ZhengGong-amd]

Finding 3 is real and worth fixing. Finding 2 is fixed in the wrong direction, and
Finding 3 is carried over the wrong channel — most of the new code exists to patch
that choice. Please redraw the two mechanisms below rather than adding guards; the
watchdog business code should come out roughly half its current size.

1. One bit, one signal. "Was this SIGTERM the supervisor's?" is one bit, and
SignalDrain already receives it: the bytes it reads off the wakeup pipe are the
signal numbers (signals.py:86), and __init__ already takes an arbitrary
signals: Iterable[int]. Add SIGHUP to STOP_SIGNALS, publish
self.received.update(data) next to self.requested.set(), have the supervisor
os.kill(pid, signal.SIGHUP), and let the coordinator read
signal.SIGHUP in drain.received. That deletes _signal_stop_reason's file read,
the pid comparison, the startswith(f"{WEDGED_STOP_REASON}:") match, the
write-status-before-kill ordering, and the PermissionError rollback
(_asked_unix = 0.0 / asked.pop() / re-write) that only exists because of the
early bookkeeping. It also closes a hole the current form has: stop_asked is
sticky, so after the supervisor has asked once, an operator's own kill is
classified supervisor_restart_requested and the session gets relaunched — the
"manual SIGTERM terminal handling" the description claims to preserve. The new
tests cover empty/mismatched/malformed status, not that case. As a side effect
this also stops the coordinator depending on the supervisor's file format and on
the wording of a log string, which inverts a dependency the docs call out as
one-way.

2. Derive the terminal markers from the monitor, don't enumerate them by hand.
is_terminal_session() reads three things: reports/final.md (:149),
state.phase == "CLOSE" (:160), and stop_reason ∈ STOP_REASON_VOCAB (:178). It
does not read reports/final.json. So the PR gates one marker the monitor
ignores and misses one it honours: a coordinator wedged after the phase machine
reached CLOSE still leaves phase=CLOSE behind and still won't be resumed — the
same failure, one entry condition over. The new paragraph in
environment-variables.md:696-702 states the wrong set on both ends; fix it from
the script.

3. Don't borrow stop_ts. What the next leg needs is where the previous leg's
phase segment ended; stop_ts means "the session stopped", and
sessions.py:511-513 says so in a comment and computes ended_at_utc from it. A
resumable stop still runs write_breakdown_json, so session_breakdown.json now
records an end time for a session that hasn't ended. Add leg_ended_ts, or let
_bank_previous_leg_phase_segment derive the boundary itself — either way the
stop_reason/stop_ts pairing stays intact.

4. Say where the brake is. With over = True if self._asked_unix in the DEAD
branch, and _end_attempted allowing exactly one _end that returns False while
the process is still up, supervisor_tick_stalled and supervisor_coordinator_died
become nearly unreachable — both are still in STOP_REASON_VOCAB. That terminal
record was the only thing stopping wedge → restart → wedge. If watchdog restarts
are now non-terminal, the restart count has to be bounded somewhere: carry it in
status.json and go terminal past a limit.

5. Finding 2: fix the root cause, not the detector. min(tick_age, heartbeat_age) fires only when both are stale, i.e. only when the loop is blocked
in sync code — which running(pid) already almost covers. The common hang is an
await that never returns; the pulse stays fresh through it and WEDGED becomes
unreachable. The false positive's cause is that inline tick steps are bounded by
the session: _await_within_session_bound uses
remaining = self._seconds_until_session_bound(). It already takes stage — give
it a per-stage cap (min(stage_cap, session_remaining), role turns at the
backend's call_timeout_s). That removes the false positive and fixes a real bug:
one step may currently consume the whole run. If you want a second signal instead,
use in-flight task/lease progress (resource_lock.heartbeat_at), which is evidence
of work rather than of a turning loop. Either way, the five-line rationale on
DEFAULT_TICK_STALL_SEC should be replaced by a new argument, not deleted — and
if the detector keeps a composite age, Observation should carry heartbeat_age
so status.json records the input the verdict was made on, not half of it.

6. Collapse what's left. One if not resumable_stop: around the close
sequence and the T4 hook (currently two adjacent guards, plus a third in stop()).
Drop _terminal_reports_enabled — one call site, one comparison. Drop
getattr(state, "close_sequence_done", False); state is a SharedState and your
own test constructs the field. Log the degradation in whatever replaces
_signal_stop_reason: falling back to a terminal stop because a file was
unreadable is exactly the case that needs a line, and the supervisor side already
logs its refusals. return stop_reason if stop_reason == SUPERVISOR_RESTART_REASON else … says "if it is X return X". And final.md moved from after
write_breakdown_json to before it — if that's intended, say why and pin it;
otherwise restore the order.

7. Split the PR. The description says "Findings 2 and 3 only"; the diff carries
five commits and seven unrelated files. packaging (already a module-level import
at base) and the _agentx_variant_timeout extraction (good — it replaces an
except ImportError fallback with a real dependency cut) are each fine on their
own. Two are not, and need their own review:

• install_baremetal.sh: checkout -q "$SGLANG_REF" is correct when the ref is a
  SHA, which is the only case the new test builds. When it's a branch name, this
  checks out a possibly stale local branch, where FETCH_HEAD was guaranteed to be
  what was just fetched. Resolve FETCH_HEAD to a SHA, or cover the branch case.
• pre-release probe: moving the real execution to a baremetal leg is defensible,
  but keep the pytest test that actually runs the probe (it skips when the
  framework is absent) instead of replacing it with bootstrap_script.index(...)
  ordering assertions on source text. Also note set -a; . "$envf"; set +a exports
  the whole env file into everything the leg runs afterwards, and a probe failure
  aborts the leg without going through log.

Finally: no CHANGELOG entry, and SKILL.md (the one place that mentions the
supervisor) doesn't learn that a watchdog restart is not a session outcome.