MOR-1191: drive the TX safety supervisor so the watchdog can fire

[morozsm]

Urgent, blocks MOR-1016. 3 files, 196 net LOC.

The defect

TxSafetySupervisor.tick() had no caller in src/. It is the sole path to two things:

• firing the BACKEND_MAX_KEY_DOWN watchdog — _watchdog_deadline is read nowhere else;
• driving _service_release(force=False), the timed retry of a failed OFF — every other release call site passes force=True.

So a rig keyed through the managed path stayed keyed indefinitely, and a failed de-key set a retry_due that nothing ever came back for. The machinery was correct the whole time; nothing drove it.

watchdog_enabled           : True
watchdog_deadline (t=1000) : 1180.0
after +3600s, writes seen  : []          <- an hour past the deadline
effects when tick() called : ['ProviderAttempt:write_off']
release reason after tick  : backend_max_key_down

The design

The ticker lives in ManagedRadioRuntime and is armed lazily by request_on.

Deliberately not a run()/start() hook in the AcquisitionScheduler shape: an AST scan finds zero ManagedRadioRuntime constructions in src/, so a start hook would be one more thing MOR-1016 could forget — and forgetting it silently restores exactly this defect. Arming inside request_on is complete because _Lease is constructed at exactly one site, inside request_on, confirmed by AST rather than grep.

The loop retires itself once the lease is gone. Measured:

idle target  = 0 reducer calls, 0 ticker tasks    <- free in the strong sense: no task exists
keyed target = 3.5 reducer calls/s, 0 wire writes
watchdog deadline overshoot = 8 ms (interval 250 ms)

Interval 0.25 s = retry_schedule_seconds[0], the finest granularity the supervisor's own retry schedule can express; a longer interval would make the first _schedule_retry step unrepresentable.

Locking: only _lifecycle_lock, and only around the reducer call. _lifecycle_change guards provider identity, which a tick never changes, and waiting on it would park the watchdog behind a retirement — exactly when a keyed rig most needs it. Effects are serviced outside the lock.

watchdog_enabled now means configured and driven: only a real tick() sets _driven. The field can no longer advertise a watchdog nothing runs, which is how this defect stayed invisible.

Evidence

Independent verification at max effort, all three files sha256-pinned and unchanged. Every experiment ran in an isolated git archive tree with its own venv, each asserting its rigplane resolution before use.

The load-bearing question — can a pending release outlive the ticker? The loop retires on lease_id is None, so if the lease could be cleared while a release was still retrying, acceptance criterion 2 would silently not hold. AST scan found _lease written at exactly three sites; _release is cleared only by the same chained assignment that clears _lease. Then run, not rested on:

probe	result
invariant asserted after every reducer entry point, incl. 6 failed-retry rounds	never stranded
key → fail OFF → advance 0.25 s	['ptt(off)','ptt(off)','read_ptt'], ends IDLE — the retry reaches the wire
compound: watchdog fires, then its own OFF fails 3×	the same ticker carries it through the 0.25/1.0/2.0 backoff → 4 writes, ends IDLE, then retires

Concurrency: 60 seeded randomized interleavings of request_off / release_owner / replace_provider / shutdown against a due watchdog at randomized sub-interval offsets — 0 anomalies, 0 leftover tasks. Plus 25 operator-vs-watchdog races with never a duplicate WRITE_OFF.

One case was initially flagged — replace_provider on a due watchdog emitting two OFFs — and the reviewer ran the same scenario in a pure-base tree with no ticker at all and got the identical sequence. Pre-existing MOR-1013 Slice 6 recovery semantics; their assertion was too strict, not the code.

12 mutations, 9 killed. Both mutations I specifically demanded are killed: tick called but the transition dropped, and the ticker never retires. So the tests catch a regression, not merely the absence of the feature.

Red-before stage 2 verified independently: with base source plus only the ctor knob (no loop, no arming, no cancel — grep-confirmed), all four tests fail on the defect itself.

gate	3.11.13	3.13.5
pytest tests/ --ignore=tests/integration — base	8534	8534
same — head	8538	8538
ruff check / format --check	clean	clean
lint-imports	5 kept, 0 broken	5 kept, 0 broken
mypy src/	15 errors, byte-identical list to base	identical

New test file run 80× across both interpreters, 20 of them under load average 4.1 — 0 failures. Not timing-fragile.

Inert for shipped behaviour, provably: ManagedRadioRuntime has zero construction sites in src/; TxSafetySupervisor is constructed in exactly one place, inside it; watchdog_enabled has zero consumers in src/ and TxSafetySnapshot never reaches a wire or API payload.

The author's disclosed non-kill, and what the reviewer did with it

The author reported mutation M3 — ticker ignores the _shutdown_pending fence — as SURVIVED, could not kill it, kept the guard on judgement, and reported 6/7 rather than claiming 7/7.

The reviewer reproduced the author's hypothesised scenario and settled it: with a service that swallows CancelledError and the ticker parked inside _service_effects, removing the fence makes shutdown() never return — the loop survives its cancel, lease_id is still non-None, and _complete_shutdown's gather hangs. The fence is not redundant. The author's "could not kill it" was a gap in the test, not a property of the code.

Follow-up filed — MOR-1194, blocking MOR-1016

The reviewer's findings, none reachable today:

1. _tick_task dangles after any external cancel. The loop catches CancelledError and returns without clearing _tick_task, so request_on never re-arms and the watchdog is permanently dead for that runtime — while watchdog_enabled still reports True. Only _complete_shutdown cancels in-tree and it clears first, so this is latent — but it is the exact failure mode this feature exists to prevent, which is why it blocks MOR-1016 rather than sitting in Backlog.
2. Adopt the reviewer's shutdown-hang test, which kills M3.
3. The interval is unpinned — sleep(0) passes all four tests, so a busy-spin regression would ship silently.
4. Servicing effects inside the lock passes; it does not deadlock but does stall provider retirement. The out-of-lock choice is right and materially better, just unpinned.

Hardware boundary

Software only. No hardware was run and no hardware claim is made. MOR-1033 remains the FTX-1 physical acceptance gate.

Linear: MOR-1191

[morozsm]

Agent Review: PASS

Independent non-author verification at max effort. All three files sha256-pinned before and after; unchanged, and the commit introduced no delta:

f806afbc392ec1174f3bd6a2ce9d1c8c9b067391446e23f7c4dfc41256435265  src/rigplane/core/tx_safety.py
512cd797b6a635959794a2632f3daa55a278f3033f9ed2db7f5691da3ba3eaab  src/rigplane/runtime/managed_radio_runtime.py
0e537cb3b21b0343dae788df92bfa7f767d0f7ebf341bf2f9e1a47427bc76cca  tests/test_managed_tx_watchdog_ticker.py

Every experiment ran in one of three isolated git archive trees — sandbox, pure-base, red-before — each with its own venv asserting its rigplane resolution before use.

The load-bearing question — can a pending release outlive the ticker? No.

The loop retires on lease_id is None, so if the lease could be cleared while a release was still retrying, acceptance criterion 2 would silently not hold and no test would necessarily catch it.

AST scan found _lease written at exactly three sites; _release is cleared only by the same chained assignment that clears _lease, and its one caller is a boundary-accepted OFF. Then run rather than rested on:

probe	result
invariant asserted after every reducer entry point, incl. 6 failed-retry rounds	never stranded
tick() with no lease	NOOP, zero effects — retirement loses nothing
key → fail OFF → advance 0.25 s	['ptt(off)','ptt(off)','read_ptt'], ends IDLE — the retry reaches the wire
compound, not covered by the change's own tests: watchdog fires, then its own OFF fails 3×	the same ticker carries it through the 0.25/1.0/2.0 backoff → 4 writes, ends IDLE, then retires

Arming, re-arming, shutdown

_Lease is constructed at exactly one site, confirmed by AST — no setattr, no replace, no deserialisation. Re-keying after retirement arms exactly one ticker; re-keying while the old ticker sleeps arms no second one, and the old ticker adopts the new lease and still fires. A BUSY request_on spawns nothing.

Shutdown cancels cleanly mid-sleep and mid-_service_effects. The ticker cannot cancel itself, and the async with self._lifecycle_lock body contains no await, so cancellation can never strand the lock.

The author's disclosed non-kill — settled, in the author's favour

The author reported mutation M3 (ticker ignores the _shutdown_pending fence) as surviving, said they could not kill it, kept the guard on judgement, and reported 6/7 rather than claiming 7/7.

The reviewer reproduced the author's hypothesised scenario and confirmed it: with a service that swallows CancelledError and the ticker parked inside _service_effects, removing the fence makes shutdown() never return — the loop survives its cancel, lease_id is still non-None, so it ticks forever and _complete_shutdown's gather hangs. The fence is not redundant with the cancel. The "could not kill it" was a gap in the test, not a property of the code.

Concurrency

60 seeded randomized interleavings of request_off / release_owner / replace_provider / shutdown / concurrent-both, each landing on a due watchdog at randomized sub-interval offsets: 0 anomalies, 0 leftover ticker tasks. Plus 25 operator-vs-watchdog races with never a duplicate WRITE_OFF, and no WRITE_ON during teardown.

One case was initially flagged — replace_provider on a due watchdog emitting two OFFs. The reviewer ran the identical scenario in a pure-base tree with no ticker at all and got the identical sequence: pre-existing MOR-1013 Slice 6 recovery semantics, both writes in the fail-safe direction. Their assertion was too strict, not the code. Worth noting as an example of checking a suspicion against base before attributing it.

Cost, measured with a real clock

idle target  = 0 reducer calls, 0 ticker tasks   <- free in the strong sense: no task exists
keyed target = 3.5 reducer calls/s, 0 wire writes
watchdog deadline overshoot = 8 ms on a 250 ms interval

retry_schedule_seconds[0] == 0.25 confirmed at tx_safety.py:221.

Vacuity and red-before

12 mutations, 9 killed. Both mutations the brief specifically demanded are killed — tick called but the transition dropped, and the ticker never retires. So these tests catch a regression, not merely the absence of the feature.

Red-before stage 2 verified independently: with base source plus only the ctor knob and the _tick_task attribute — no loop, no arming, no cancel, grep-confirmed — all four tests fail on the defect itself, not on a missing kwarg.

Gates — run by the reviewer, both interpreters

gate	3.11.13	3.13.5
pytest tests/ --ignore=tests/integration — base	8534	8534
same — head	8538	8538
ruff check / format --check (3 files)	clean	clean
lint-imports	5 kept, 0 broken	5 kept, 0 broken
mypy src/	15 errors, byte-identical list to base	identical

Exactly +4 with an unchanged skip/deselect/xfail breakdown. New test file run 80× across both interpreters, 20 of them under load average 4.1 — 0 failures.

Inert for shipped behaviour, provably

ManagedRadioRuntime has zero construction sites in src/; TxSafetySupervisor is constructed in exactly one place, inside it; watchdog_enabled has zero consumers in src/ and TxSafetySnapshot never reaches a wire or API payload. No lease ⟹ no task, no wakeups.

On the 8537-vs-8538 anomaly the author flagged

Not reproduced across four full runs, all with an identical breakdown. The reviewer did find genuine order-dependence elsewhere — tests/test_naming_parity.py yields 10 xfailed + 2 xpassed in isolation but contributes 11 + 1 in the full run, i.e. global state leaking between tests — but it is identical at base and head, and an xfail↔xpass flip cannot move the passed count. Filed separately as MOR-1195; structurally independent of this change.

Non-blocking findings — all filed as MOR-1194, blocking MOR-1016

1. _tick_task dangles after any external cancel. The loop catches CancelledError and returns without clearing it, so request_on never re-arms and the watchdog is permanently dead for that runtime — while watchdog_enabled still reports True. Only _complete_shutdown cancels in-tree and it clears first, so this is latent, but it is the exact failure mode this feature exists to prevent. The two-line fix (finally:) would have fit in the budget; its test would not, and shipping a safety fix without its test is what this programme has repeatedly declined to do.
2. Adopt the reviewer's shutdown-hang test, which kills M3.
3. The interval is unpinned — sleep(0) passes all four tests.
4. Servicing effects inside the lock passes; it stalls provider retirement rather than deadlocking.

Hardware boundary

Software only. No hardware was involved and no hardware claim is made. This does not replace MOR-1033.