Repository navigation
fix: heartbeat default kills live agents; 797th manifest undoes production on re-apply - #18
Merged
Merged
Conversation
PR #17 raised `heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECS` to 600, but production never reads it. `kernel::start_heartbeat_monitor` builds its `HeartbeatConfig` as HeartbeatConfig { default_timeout_secs: self.config.heartbeat.default_timeout_secs, ..HeartbeatConfig::default() } so the `[heartbeat]` config value overrides the constant. With no `[heartbeat]` section present, that resolves to `default_heartbeat_timeout()` = 180 — and the fix from #17 is dead code on every real deployment. The result, observed on a live collector agent: a turn is allowed to run for `DEFAULT_AGENT_TURN_TIMEOUT_SECS` (600s), but the heartbeat declared the agent unresponsive at ~207s, marked it Crashed, and "recovered" it — restarting the loop and discarding the in-flight turn. An agent waiting on a slow or rate-limited model could therefore never finish a single turn. Raise the config default to 900 so it outlasts a full turn, and add a regression test asserting the invariant directly rather than pinning the literal, so the two values cannot drift apart again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 797th manifest had drifted far enough from the live cluster that re-applying it — the normal way to change anything in it — would have broken the deployment it was supposed to describe. `default_model` was still pinned to `stepfun-ai/step-3.5-flash`, which began returning 410 Gone on 2026-07-27 and took down every agent at once because it is the global default. Re-applying would have restored the dead pin. Sync it to what production actually runs and note above it that a pinned model has to be re-checked before this file is applied. The `[knowledge]` block was missing entirely. It is the controlled vocabulary the kernel enforces on the graph — without it every collector silently reverts to inventing one-off entity labels, which is what made the graph unqueryable before it was introduced. `[heartbeat]` is added here too, so the fix in this branch survives a re-apply rather than living only as a hand-edit on the cluster. Also sync `falkor-sync.yaml` with the deployed script: it was missing `has_new_artifacts()` and its call site, the guard that keeps a quiet cycle from delivering an empty "no new artifacts" report every run. The repo copy would have re-enabled that noise. Finally, commit the collector briefs. Each `context.md` is the only definition of its beat and is re-read every turn, but both existed nowhere except the PVC — a lost volume meant rewriting them from memory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nideesh1
marked this pull request as ready for review
July 31, 2026 19:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related failures, both of which stop a collector agent from ever completing a run.
1. The heartbeat default kills agents mid-turn
PR #17 raised
heartbeat::DEFAULT_UNRESPONSIVE_TIMEOUT_SECSto 600 — but production never reads it.kernel::start_heartbeat_monitor(kernel.rs:4865) builds its config as:The
[heartbeat]TOML value overrides the constant. With no[heartbeat]section present it falls back todefault_heartbeat_timeout()= 180, so #17 is dead code on every real deployment.Observed on a live collector, where a turn is allowed 600s (
DEFAULT_AGENT_TURN_TIMEOUT_SECS):Killed at 207s, in-flight work discarded, roughly three times per allowed turn — forever. An agent waiting on a slow or rate-limited model could never finish one.
default_heartbeat_timeout()180 → 900heartbeat.rsconstanttest_heartbeat_default_exceeds_agent_turn_timeoutasserts the invariant rather than the literal, so the two cannot silently drift apart again2. The 797th manifest would undo production if re-applied
deploy/797th/openfang.yamlhad drifted far enough that applying it — the normal way to change anything in it — would break the deployment it describes:default_modelstepfun-ai/step-3.5-flash— 410 Gone since 2026-07-27thinkingmachines/inkling[knowledge][heartbeat]The dead model pin is the worst of the three: it is the global default, so restoring it takes down every agent at once. That already happened once. The
[knowledge]block is the vocabulary the kernel enforces on the graph — without it collectors silently revert to inventing one-off labels.falkor-sync.yamlwas also missinghas_new_artifacts()and its call site — the guard that keeps a quiet cycle from delivering an empty "no new artifacts" report every run. Verified the repo copy now matches the deployed script byte-for-byte apart from comment wording.3. Collector briefs are now in the repo
Each collector instance is steered by a
context.mdre-read every turn (agent_context.rs::load_context_md) — no rebuild, no restart. Those files are the only definition of each beat and existed nowhere but the PVC; a lost volume meant rewriting them from memory. Addeddeploy/797th/collector-briefs/with theundergroundandremote-jobsbriefs and a README covering the report-format contract the delivery job depends on.Verification
Security Auditfails on RUSTSEC-2026-0222 (wasmtime) — a newly published advisory, not introduced here. This branch changes noCargo.tomlorCargo.lock; the diff is one Rust constant plus YAML and Markdownsync.pyextracted from the manifest andpy_compiledApplying without waiting for merge
🤖 Generated with Claude Code