Repository navigation
fix(kernel): survive restarts — don't kill live agents, don't rename hand instances - #17
Merged
Merged
Conversation
An autonomous config could make the liveness bound TIGHTER than the default, and did so for every agent using the default 30s heartbeat interval: `30 * UNRESPONSIVE_MULTIPLIER` is 60s, against a default of 180s and a turn timeout of 600s. Observed on a collector agent waiting on a rate-limited reasoning model, flapping on an exact 60-second cycle: 05:37:48 Unresponsive Running agent marked as Crashed inactive_secs=89 05:38:48 Agent recovered successfully 05:39:48 Unresponsive Running agent marked as Crashed inactive_secs=89 05:40:48 Agent recovered successfully 05:41:48 Unresponsive Running agent marked as Crashed inactive_secs=77 Each "recovery" restarts the agent loop and discards the in-flight turn, so a sweep slower than 60s could never finish — it was killed and restarted indefinitely. Waiting on the model is not being hung. Two changes: * The per-agent value now only ever RAISES the bound — `max()` against the default — so a short heartbeat interval cannot shorten the death threshold. The interval describes how often an agent checks in, not how long a turn may legitimately take. * The default moves 180 -> 600 to match DEFAULT_AGENT_TURN_TIMEOUT_SECS. An agent still inside its turn budget is by definition not dead, and the turn timeout remains the mechanism that ends an over-running turn; this value only decides when to call an agent dead. The old 180s already conceded that "browser tasks and complex LLM calls can take 1-3 minutes" — it was simply short of what a full turn is allowed. test_active_agent_detected_unresponsive derived its timings from the old 180s bound, so it now derives them from the constant instead and keeps testing "past the timeout" if the default moves again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`persist_state` never wrote `instance_name`, `load_state` never returned
it, and the restore path passed `None`:
match self.activate_hand(&hand_id, config, None) {
So every named instance came back as the unnamed one. The name decides
the agent id — `hand_instance_<instance_id>` versus the legacy id derived
from `hand_id` — and the agent name, and therefore the workspace
directory. A collector activated as "underground" returned after a deploy
as "collector-hand" with a different agent id, in a different workspace,
leaving its context.md and accumulated output orphaned. The original
agent survived as a stray with no hand registration, so the box ended up
running two collectors against one knowledge graph.
It also defeated the cron reassignment immediately below it: jobs were
migrated onto the wrong agent, so a tuned schedule (cadence, timeout)
silently reverted to the hand's defaults on every restart.
Persist the name, return it, and restore under it. `instance_name` is
read with `.get()` so state written before this field existed still
loads, as unnamed — matching today's behaviour for those entries.
Two tests: a named instance survives persist+reload, and legacy state
without the field loads as unnamed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nideesh1
added a commit
that referenced
this pull request
Jul 31, 2026
…ction on re-apply (#18) * fix(config): raise heartbeat default above the agent turn timeout PR #17 raised `heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECS` to 600, but production never reads it. `kernel::start_heartbeat_monitor` builds its `HeartbeatConfig` as HeartbeatConfig { default_timeout_secs: self.config.heartbeat.default_timeout_secs, ..HeartbeatConfig::default() } so the `[heartbeat]` config value overrides the constant. With no `[heartbeat]` section present, that resolves to `default_heartbeat_timeout()` = 180 — and the fix from #17 is dead code on every real deployment. The result, observed on a live collector agent: a turn is allowed to run for `DEFAULT_AGENT_TURN_TIMEOUT_SECS` (600s), but the heartbeat declared the agent unresponsive at ~207s, marked it Crashed, and "recovered" it — restarting the loop and discarding the in-flight turn. An agent waiting on a slow or rate-limited model could therefore never finish a single turn. Raise the config default to 900 so it outlasts a full turn, and add a regression test asserting the invariant directly rather than pinning the literal, so the two values cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(deploy/797th): stop the manifest from undoing production on re-apply The 797th manifest had drifted far enough from the live cluster that re-applying it — the normal way to change anything in it — would have broken the deployment it was supposed to describe. `default_model` was still pinned to `stepfun-ai/step-3.5-flash`, which began returning 410 Gone on 2026-07-27 and took down every agent at once because it is the global default. Re-applying would have restored the dead pin. Sync it to what production actually runs and note above it that a pinned model has to be re-checked before this file is applied. The `[knowledge]` block was missing entirely. It is the controlled vocabulary the kernel enforces on the graph — without it every collector silently reverts to inventing one-off entity labels, which is what made the graph unqueryable before it was introduced. `[heartbeat]` is added here too, so the fix in this branch survives a re-apply rather than living only as a hand-edit on the cluster. Also sync `falkor-sync.yaml` with the deployed script: it was missing `has_new_artifacts()` and its call site, the guard that keeps a quiet cycle from delivering an empty "no new artifacts" report every run. The repo copy would have re-enabled that noise. Finally, commit the collector briefs. Each `context.md` is the only definition of its beat and is re-read every turn, but both existed nowhere except the PVC — a lost volume meant rewriting them from memory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: openfang <scaling.workflow.systems@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two restart/liveness bugs found while running a collector agent in production. Both destroy work silently.
1. Agents declared dead while waiting on the model
An
AutonomousConfigcould make the liveness bound tighter than the default, and did so for every agent using the default 30s heartbeat interval:A turn is allowed 600 seconds; the agent was declared dead after 60. Observed live, flapping on an exact 60-second cycle:
Each "recovery" restarts the loop and discards the in-flight turn, so any sweep whose model call exceeds 60s could never finish. Waiting on the model is not being hung. Likely cause of previously-seen sweeps that burned their whole iteration budget and delivered nothing, and of scheduled jobs hitting their turn timeout.
Fix. The per-agent value now only ever raises the bound (
max()against the default) — a heartbeat interval describes how often an agent checks in, not how long a turn may take. The default moves 180 → 600 to matchDEFAULT_AGENT_TURN_TIMEOUT_SECS; an agent inside its turn budget is by definition not dead, and the turn timeout remains what ends an over-running turn.Trade-off: a genuinely hung agent now takes up to 10 minutes to detect instead of 1. That is the right side to err on — the current behaviour destroys live work every 60 seconds, while a hung agent is still caught, just later, with auto-recovery unchanged.
2. Named hand instances lose their name on restart
persist_statenever wroteinstance_name,load_statenever returned it, and the restore path passedNone:The name decides the agent id (
hand_instance_<instance_id>vs the legacy id derived fromhand_id), the agent name, and therefore the workspace directory. A collector activated asundergroundcame back after a deploy ascollector-hand, under a different agent id, in a different workspace — orphaning itscontext.mdand accumulated output. The original agent survived as a stray with no hand registration, so the box ended up running two collectors against one knowledge graph, both delivering reports.It also defeated the cron reassignment immediately below it: jobs were migrated onto the wrong agent, so a tuned schedule (cadence, timeout) silently reverted to hand defaults on every restart. Observed exactly that — a 3-hourly/2400s schedule reverted to 6-hourly with no timeout, and then failed with
error: timed out after 600s.Fix. Persist the name, return it, restore under it. Read with
.get()so state written before the field existed still loads, as unnamed — matching today's behaviour for those entries.Tests
test_autonomous_config_never_shortens_timeout— the default 30s interval no longer drags the bound below the default; a generous interval still raises it.test_instance_name_survives_persist_reload— a named instance comes back named.test_load_state_without_instance_name_is_unnamed— legacy state still loads.test_active_agent_detected_unresponsivehad its timings calibrated to the old 180s bound (300s inactivity was "past timeout" then, but is inside 600s now), so it would have silently started asserting the wrong thing. It now derives them fromDEFAULT_UNRESPONSIVE_TIMEOUT_SECS.test_heartbeat_config_defaultlikewise asserts against the constant.Simulated the threshold-sensitive cases against the new bound before submitting:
cargo fmt --all --checkclean. Not compiled locally — this host is a cluster node and cargo OOM-killed kube-apiserver on it previously.🤖 Generated with Claude Code