Skip to content

fix(kernel): survive restarts — don't kill live agents, don't rename hand instances - #17

Merged
Nideesh1 merged 2 commits into
mainfrom
fix/heartbeat-idle-flap
Jul 30, 2026
Merged

Nideesh1 merged 2 commits into
mainfrom
fix/heartbeat-idle-flap

Conversation

@Nideesh1

@Nideesh1 Nideesh1 commented Jul 29, 2026 •

Copy link
Copy Markdown

Two restart/liveness bugs found while running a collector agent in production. Both destroy work silently.


1. Agents declared dead while waiting on the model

An AutonomousConfig could make the liveness bound tighter than the default, and did so for every agent using the default 30s heartbeat interval:

heartbeat_interval_secs = 30   (agent.rs, default)
UNRESPONSIVE_MULTIPLIER = 2
→ 60s liveness window

HeartbeatConfig::default_timeout_secs = 180
DEFAULT_AGENT_TURN_TIMEOUT_SECS       = 600

A turn is allowed 600 seconds; the agent was declared dead after 60. Observed live, flapping on an exact 60-second cycle:

05:37:48 Unresponsive Running agent marked as Crashed  inactive_secs=89
05:38:18 Auto-recovering crashed agent (attempt 1/3)
05:38:48 Agent recovered successfully
05:39:48 Unresponsive Running agent marked as Crashed  inactive_secs=89
05:40:48 Agent recovered successfully
05:41:48 Unresponsive Running agent marked as Crashed  inactive_secs=77

Each "recovery" restarts the loop and discards the in-flight turn, so any sweep whose model call exceeds 60s could never finish. Waiting on the model is not being hung. Likely cause of previously-seen sweeps that burned their whole iteration budget and delivered nothing, and of scheduled jobs hitting their turn timeout.

Fix. The per-agent value now only ever raises the bound (max() against the default) — a heartbeat interval describes how often an agent checks in, not how long a turn may take. The default moves 180 → 600 to match DEFAULT_AGENT_TURN_TIMEOUT_SECS; an agent inside its turn budget is by definition not dead, and the turn timeout remains what ends an over-running turn.

Trade-off: a genuinely hung agent now takes up to 10 minutes to detect instead of 1. That is the right side to err on — the current behaviour destroys live work every 60 seconds, while a hung agent is still caught, just later, with auto-recovery unchanged.

2. Named hand instances lose their name on restart

persist_state never wrote instance_name, load_state never returned it, and the restore path passed None:

match self.activate_hand(&hand_id, config, None) {

The name decides the agent id (hand_instance_<instance_id> vs the legacy id derived from hand_id), the agent name, and therefore the workspace directory. A collector activated as underground came back after a deploy as collector-hand, under a different agent id, in a different workspace — orphaning its context.md and accumulated output. The original agent survived as a stray with no hand registration, so the box ended up running two collectors against one knowledge graph, both delivering reports.

It also defeated the cron reassignment immediately below it: jobs were migrated onto the wrong agent, so a tuned schedule (cadence, timeout) silently reverted to hand defaults on every restart. Observed exactly that — a 3-hourly/2400s schedule reverted to 6-hourly with no timeout, and then failed with error: timed out after 600s.

Fix. Persist the name, return it, restore under it. Read with .get() so state written before the field existed still loads, as unnamed — matching today's behaviour for those entries.


Tests

  • test_autonomous_config_never_shortens_timeout — the default 30s interval no longer drags the bound below the default; a generous interval still raises it.
  • test_instance_name_survives_persist_reload — a named instance comes back named.
  • test_load_state_without_instance_name_is_unnamed — legacy state still loads.

test_active_agent_detected_unresponsive had its timings calibrated to the old 180s bound (300s inactivity was "past timeout" then, but is inside 600s now), so it would have silently started asserting the wrong thing. It now derives them from DEFAULT_UNRESPONSIVE_TIMEOUT_SECS. test_heartbeat_config_default likewise asserts against the constant.

Simulated the threshold-sensitive cases against the new bound before submitting:

idle_agent_skipped            inactive=300s -> skipped (never active)  OK
active_detected_unresponsive  inactive=660s -> unresponsive            OK
active_within_timeout_ok      inactive= 10s -> responsive              OK

cargo fmt --all --check clean. Not compiled locally — this host is a cluster node and cargo OOM-killed kube-apiserver on it previously.

🤖 Generated with Claude Code

scaling-workflow-systems and others added 2 commits July 29, 2026 05:48
An autonomous config could make the liveness bound TIGHTER than the
default, and did so for every agent using the default 30s heartbeat
interval: `30 * UNRESPONSIVE_MULTIPLIER` is 60s, against a default of
180s and a turn timeout of 600s.

Observed on a collector agent waiting on a rate-limited reasoning model,
flapping on an exact 60-second cycle:

  05:37:48 Unresponsive Running agent marked as Crashed  inactive_secs=89
  05:38:48 Agent recovered successfully
  05:39:48 Unresponsive Running agent marked as Crashed  inactive_secs=89
  05:40:48 Agent recovered successfully
  05:41:48 Unresponsive Running agent marked as Crashed  inactive_secs=77

Each "recovery" restarts the agent loop and discards the in-flight turn,
so a sweep slower than 60s could never finish — it was killed and
restarted indefinitely. Waiting on the model is not being hung.

Two changes:

* The per-agent value now only ever RAISES the bound — `max()` against
  the default — so a short heartbeat interval cannot shorten the death
  threshold. The interval describes how often an agent checks in, not
  how long a turn may legitimately take.

* The default moves 180 -> 600 to match DEFAULT_AGENT_TURN_TIMEOUT_SECS.
  An agent still inside its turn budget is by definition not dead, and
  the turn timeout remains the mechanism that ends an over-running turn;
  this value only decides when to call an agent dead. The old 180s
  already conceded that "browser tasks and complex LLM calls can take
  1-3 minutes" — it was simply short of what a full turn is allowed.

test_active_agent_detected_unresponsive derived its timings from the old
180s bound, so it now derives them from the constant instead and keeps
testing "past the timeout" if the default moves again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`persist_state` never wrote `instance_name`, `load_state` never returned
it, and the restore path passed `None`:

    match self.activate_hand(&hand_id, config, None) {

So every named instance came back as the unnamed one. The name decides
the agent id — `hand_instance_<instance_id>` versus the legacy id derived
from `hand_id` — and the agent name, and therefore the workspace
directory. A collector activated as "underground" returned after a deploy
as "collector-hand" with a different agent id, in a different workspace,
leaving its context.md and accumulated output orphaned. The original
agent survived as a stray with no hand registration, so the box ended up
running two collectors against one knowledge graph.

It also defeated the cron reassignment immediately below it: jobs were
migrated onto the wrong agent, so a tuned schedule (cadence, timeout)
silently reverted to the hand's defaults on every restart.

Persist the name, return it, and restore under it. `instance_name` is
read with `.get()` so state written before this field existed still
loads, as unnamed — matching today's behaviour for those entries.

Two tests: a named instance survives persist+reload, and legacy state
without the field loads as unnamed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Nideesh1 Nideesh1 changed the title fix(kernel): stop declaring agents dead while they wait on the model fix(kernel): survive restarts — don't kill live agents, don't rename hand instances Jul 29, 2026
@Nideesh1
Nideesh1 merged commit b50aea7 into main Jul 30, 2026
11 checks passed
Nideesh1 added a commit that referenced this pull request Jul 31, 2026
…ction on re-apply (#18)

* fix(config): raise heartbeat default above the agent turn timeout

PR #17 raised `heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECS` to 600, but
production never reads it. `kernel::start_heartbeat_monitor` builds its
`HeartbeatConfig` as

    HeartbeatConfig {
        default_timeout_secs: self.config.heartbeat.default_timeout_secs,
        ..HeartbeatConfig::default()
    }

so the `[heartbeat]` config value overrides the constant. With no
`[heartbeat]` section present, that resolves to `default_heartbeat_timeout()`
= 180 — and the fix from #17 is dead code on every real deployment.

The result, observed on a live collector agent: a turn is allowed to run for
`DEFAULT_AGENT_TURN_TIMEOUT_SECS` (600s), but the heartbeat declared the agent
unresponsive at ~207s, marked it Crashed, and "recovered" it — restarting the
loop and discarding the in-flight turn. An agent waiting on a slow or
rate-limited model could therefore never finish a single turn.

Raise the config default to 900 so it outlasts a full turn, and add a
regression test asserting the invariant directly rather than pinning the
literal, so the two values cannot drift apart again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(deploy/797th): stop the manifest from undoing production on re-apply

The 797th manifest had drifted far enough from the live cluster that
re-applying it — the normal way to change anything in it — would have broken
the deployment it was supposed to describe.

`default_model` was still pinned to `stepfun-ai/step-3.5-flash`, which began
returning 410 Gone on 2026-07-27 and took down every agent at once because it
is the global default. Re-applying would have restored the dead pin. Sync it to
what production actually runs and note above it that a pinned model has to be
re-checked before this file is applied.

The `[knowledge]` block was missing entirely. It is the controlled vocabulary
the kernel enforces on the graph — without it every collector silently reverts
to inventing one-off entity labels, which is what made the graph unqueryable
before it was introduced.

`[heartbeat]` is added here too, so the fix in this branch survives a re-apply
rather than living only as a hand-edit on the cluster.

Also sync `falkor-sync.yaml` with the deployed script: it was missing
`has_new_artifacts()` and its call site, the guard that keeps a quiet cycle
from delivering an empty "no new artifacts" report every run. The repo copy
would have re-enabled that noise.

Finally, commit the collector briefs. Each `context.md` is the only definition
of its beat and is re-read every turn, but both existed nowhere except the PVC
— a lost volume meant rewriting them from memory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: openfang <scaling.workflow.systems@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants