Skip to content

fix: heartbeat default kills live agents; 797th manifest undoes production on re-apply - #18

Merged
Nideesh1 merged 2 commits into
mainfrom
fix/heartbeat-config-default
Jul 31, 2026
Merged

Nideesh1 merged 2 commits into
mainfrom
fix/heartbeat-config-default

Conversation

@Nideesh1

@Nideesh1 Nideesh1 commented Jul 31, 2026 •

Copy link
Copy Markdown

Two related failures, both of which stop a collector agent from ever completing a run.

1. The heartbeat default kills agents mid-turn

PR #17 raised heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECS to 600 — but production never reads it. kernel::start_heartbeat_monitor (kernel.rs:4865) builds its config as:

HeartbeatConfig {
    default_timeout_secs: self.config.heartbeat.default_timeout_secs,
    ..HeartbeatConfig::default()
}

The [heartbeat] TOML value overrides the constant. With no [heartbeat] section present it falls back to default_heartbeat_timeout() = 180, so #17 is dead code on every real deployment.

Observed on a live collector, where a turn is allowed 600s (DEFAULT_AGENT_TURN_TIMEOUT_SECS):

16:47:18  shell_exec ok
16:51:02  Unresponsive Running agent marked as Crashed, inactive_secs=207
16:51:32  Auto-recovering crashed agent (attempt 1/3)
16:52:02  Agent recovered successfully

Killed at 207s, in-flight work discarded, roughly three times per allowed turn — forever. An agent waiting on a slow or rate-limited model could never finish one.

  • default_heartbeat_timeout() 180 → 900
  • doc comment recording that this is the value production uses, not the heartbeat.rs constant
  • new test_heartbeat_default_exceeds_agent_turn_timeout asserts the invariant rather than the literal, so the two cannot silently drift apart again

2. The 797th manifest would undo production if re-applied

deploy/797th/openfang.yaml had drifted far enough that applying it — the normal way to change anything in it — would break the deployment it describes:

manifest had live cluster
default_model stepfun-ai/step-3.5-flash — 410 Gone since 2026-07-27 thinkingmachines/inkling
[knowledge] absent 11 entity types, 15 relations
[heartbeat] absent needed, see above

The dead model pin is the worst of the three: it is the global default, so restoring it takes down every agent at once. That already happened once. The [knowledge] block is the vocabulary the kernel enforces on the graph — without it collectors silently revert to inventing one-off labels.

falkor-sync.yaml was also missing has_new_artifacts() and its call site — the guard that keeps a quiet cycle from delivering an empty "no new artifacts" report every run. Verified the repo copy now matches the deployed script byte-for-byte apart from comment wording.

3. Collector briefs are now in the repo

Each collector instance is steered by a context.md re-read every turn (agent_context.rs::load_context_md) — no rebuild, no restart. Those files are the only definition of each beat and existed nowhere but the PVC; a lost volume meant rewriting them from memory. Added deploy/797th/collector-briefs/ with the underground and remote-jobs briefs and a README covering the report-format contract the delivery job depends on.

Verification

  • CI: build / test / clippy / format green on all three platforms
  • Security Audit fails on RUSTSEC-2026-0222 (wasmtime) — a newly published advisory, not introduced here. This branch changes no Cargo.toml or Cargo.lock; the diff is one Rust constant plus YAML and Markdown
  • YAML + TOML of both manifests parsed and asserted against the live cluster values
  • sync.py extracted from the manifest and py_compiled
  • Not compiled locally: the only available host is a k8s control-plane node where cargo has previously OOM-killed kube-apiserver

Applying without waiting for merge

[heartbeat]
default_timeout_secs = 900

🤖 Generated with Claude Code

scaling-workflow-systems and others added 2 commits July 31, 2026 17:00
PR #17 raised `heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECS` to 600, but
production never reads it. `kernel::start_heartbeat_monitor` builds its
`HeartbeatConfig` as

    HeartbeatConfig {
        default_timeout_secs: self.config.heartbeat.default_timeout_secs,
        ..HeartbeatConfig::default()
    }

so the `[heartbeat]` config value overrides the constant. With no
`[heartbeat]` section present, that resolves to `default_heartbeat_timeout()`
= 180 — and the fix from #17 is dead code on every real deployment.

The result, observed on a live collector agent: a turn is allowed to run for
`DEFAULT_AGENT_TURN_TIMEOUT_SECS` (600s), but the heartbeat declared the agent
unresponsive at ~207s, marked it Crashed, and "recovered" it — restarting the
loop and discarding the in-flight turn. An agent waiting on a slow or
rate-limited model could therefore never finish a single turn.

Raise the config default to 900 so it outlasts a full turn, and add a
regression test asserting the invariant directly rather than pinning the
literal, so the two values cannot drift apart again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 797th manifest had drifted far enough from the live cluster that
re-applying it — the normal way to change anything in it — would have broken
the deployment it was supposed to describe.

`default_model` was still pinned to `stepfun-ai/step-3.5-flash`, which began
returning 410 Gone on 2026-07-27 and took down every agent at once because it
is the global default. Re-applying would have restored the dead pin. Sync it to
what production actually runs and note above it that a pinned model has to be
re-checked before this file is applied.

The `[knowledge]` block was missing entirely. It is the controlled vocabulary
the kernel enforces on the graph — without it every collector silently reverts
to inventing one-off entity labels, which is what made the graph unqueryable
before it was introduced.

`[heartbeat]` is added here too, so the fix in this branch survives a re-apply
rather than living only as a hand-edit on the cluster.

Also sync `falkor-sync.yaml` with the deployed script: it was missing
`has_new_artifacts()` and its call site, the guard that keeps a quiet cycle
from delivering an empty "no new artifacts" report every run. The repo copy
would have re-enabled that noise.

Finally, commit the collector briefs. Each `context.md` is the only definition
of its beat and is re-read every turn, but both existed nowhere except the PVC
— a lost volume meant rewriting them from memory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Nideesh1 Nideesh1 changed the title fix(config): raise heartbeat default above the agent turn timeout fix: heartbeat default kills live agents; 797th manifest undoes production on re-apply Jul 31, 2026
@Nideesh1
Nideesh1 marked this pull request as ready for review July 31, 2026 19:24
@Nideesh1
Nideesh1 merged commit 2b084a2 into main Jul 31, 2026
10 of 11 checks passed
Nideesh1 pushed a commit that referenced this pull request Aug 1, 2026
The branch was cut from the pre-#20 worktree branch, so its merge base was
#18 and GitHub rendered the already-merged driver-timeout changes as part of
this PR's diff. Merging main advances the merge base so the PR shows only
what it actually changes: the remote-jobs collector brief.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants