Skip to content

Commit bbaf5ff

Browse files
ralyodioclaude
andauthored
feat(herd): heartbeat liveness and an immutable run record (PRD 0019 P0) (#542)
R2. A run moshcode started records a heartbeat, and herd state is read from that before anything else. Output pattern matching stays, as a fallback, and is now labelled as inference rather than as fact. Every answer from sessionState() carries `confidence`. `known` when the run, the engine hook or the process itself said so. `inferred` when a regular expression looked at a terminal and decided. The roster marks an inferred state with a trailing `?`. Both tiers are permanent: heartbeats exist only for runs moshcode starts, so a pane somebody launched by hand is inferred forever and says so. The beat rides `moshcode herd report`, which was already the one socket any process with $MOSHCODE_HERD_NAME can call and already what the engine hooks are wired to. No second channel, and nothing new to install. When $MOSHCODE_RUN names a run, the same report also writes the run's beat. R3. A run record is immutable and versioned: an immutable `wx` header under $OPENFLEET_HOME/fleets/<fleet>/runs/ carrying record_version, inputs, the engine and the model, and run.start, run.step, run.end lines on the fleet's existing ledger. It extends the OpenFleet ledger and opens no second store, so `moshcode fleet log` reads a run with no new tool. Files, not SQLite. PRD 0016 chose no daemon, no database, one host on purpose; the record is append-only by nature, which is what a JSONL ledger is already good at; other tools read these files; and the search R4 wants is over hundreds of runs on one box. The full reasoning is written into the PRD as a resolution. Retention ships with the record, because an immutable record grows forever. `moshcode fleet gc` keeps the newest 20 ended runs per working directory and drops anything ended over 30 days ago, with --dry-run to preview. Per directory rather than globally, so one busy checkout cannot evict every other one. A run with no run.end is never a candidate at any age, only this host's ledger is rewritten, and only run.* lines are dropped. R12. No em dashes in anything added here. Short sentences. New: src/run-record.mjs, `moshcode fleet beat`, `moshcode fleet gc`. Deliberately not here: R4 search, R5 DAGs, R6 doctor, R7 precedence, R8 .agents/skills, and nothing about the credential vault, which is PRD 0018 R8. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 172f930 commit bbaf5ff

14 files changed

Lines changed: 1742 additions & 42 deletions

‎prd/0019-take-what-swamp-got-right.md‎

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -99,9 +99,15 @@ for leaving precedence undocumented.
9999
landing directory searched. Done in this change.
100100
- R2 [P0] Heartbeat liveness. A moshcode-started run records a heartbeat, and herd state
101101
is derived from it first. Output pattern matching stays as a fallback for engines that
102-
cannot report, and is labelled as inference rather than fact.
102+
cannot report, and is labelled as inference rather than fact. Done: `moshcode fleet
103+
beat` writes it, the engine's lifecycle hook calls it, `sessionState` reads it ahead of
104+
the hook and the screen, and every answer carries `confidence`, `known` or `inferred`.
105+
The roster marks an inferred state with a trailing `?`.
103106
- R3 [P0] A run record is immutable and versioned: inputs, the engine and model, every
104107
step, and outputs. It extends the OpenFleet ledger rather than opening a second store.
108+
Done: an immutable `wx` header under `fleets/<fleet>/runs/` carrying `record_version`,
109+
and `run.start`, `run.step`, `run.end` on the fleet's existing ledger. See the resolved
110+
storage question below. Retention ships with it as `moshcode fleet gc`.
105111
- R4 [P1] Run history is searchable from the CLI, the TUI and the MCP bridge, per the
106112
house agent-surfaces rule. Any dashboard is hqtui, not a localhost web page.
107113
- R5 [P1] Swarm pieces declare dependencies, and the runner derives concurrency from the
@@ -158,16 +164,63 @@ interactive session, so it lives with railway, gh and supabase instead.
158164
transitional.
159165
- An immutable run record grows without bound. It needs a retention story before it
160166
ships, not after. omp's `gc` with per-directory retention is the shape to copy.
167+
ANSWERED 2026-09-25: `moshcode fleet gc` keeps the newest 20 ended runs per working
168+
directory and drops anything ended more than 30 days ago, with `--dry-run` to preview.
169+
Per directory rather than globally, so one busy checkout cannot evict every other one.
170+
A run with no `run.end` is never a candidate at any age. Only this host's `ledger.jsonl`
171+
is rewritten, never a `ledger.<host>.jsonl` copied in from a box running its own
172+
retention. And only `run.*` lines are dropped: `member.start`, `swarm.end` and
173+
`ceiling.refuse` are the spec's record of what agents were allowed to do, and no
174+
retention policy of ours decides those have expired.
161175
- A DAG is more expressive than the current phase list, which means a planner can now
162176
author a cycle or a deadlock. The runner must reject a cyclic graph up front with the
163177
cycle named.
164178
- The license boundary is real and easy to cross by accident, particularly for anyone
165179
reading swamp's YAML schemas while implementing R5 or R11. Requirements here describe
166180
behaviour on purpose and name no swamp file. If an implementation needs their schema,
167181
that is the moment to stop and take legal advice instead.
168-
- Open question: does the run record belong in SQLite like swamp's, or in the JSON and
169-
append-only files OpenFleet already uses? SQLite buys search cheaply. Files keep the
170-
"no daemon, no database, one host" property PRD 0016 chose deliberately. Deciding this
171-
is the first task, because R3 and R4 both depend on it.
182+
- RESOLVED 2026-09-25: the run record lives in the files OpenFleet already writes, not
183+
in SQLite. Implemented in `src/run-record.mjs`.
184+
185+
Four reasons, in the order they decided it.
186+
187+
PRD 0016 chose "no daemon, no database, one host" on purpose and named it a goal, not
188+
an accident of what was easy. A database is the one change that takes that property
189+
away, and nothing in R3 or R4 needs it taken away. Reversing a deliberate design
190+
decision wants a concrete reason and there was not one.
191+
192+
The record is append-only by nature. What a run produces is a start, a sequence of
193+
steps and an end, written once each and never updated. That is the exact shape a JSONL
194+
ledger is already good at and the shape a relational store is least rewarded for. The
195+
ledger also already carries `member.start` and `swarm.end` for the same runs, so
196+
putting the steps anywhere else would mean two stores to read and two clocks to merge
197+
for one question.
198+
199+
Other tools read these files. `logicsrc fleet` reads the same ledger, and so does a
200+
shell script, and so does a person with `cat` on a box where moshcode is not
201+
installed. That reach survives only while the files stay plain. A SQLite file is
202+
readable by anything with the library and by nothing without it.
203+
204+
The search R4 wants is small. The volume here is hundreds of runs on one host, not
205+
millions, and a full scan of a JSONL ledger at that size is faster than the process
206+
start that would open a database. If a box ever holds enough runs for that to stop
207+
being true, an index derived from the files is an addition rather than a migration,
208+
because the files would still be the source of truth.
209+
210+
What the decision costs, stated rather than hidden: no transactions, so a torn line is
211+
skipped rather than rolled back, which is what `readLedger` already does; no query
212+
language, so R4 writes its own filters; and a retention policy that has to rewrite a
213+
file rather than issue a delete, which `moshcode fleet gc` does with a write and a
214+
rename. None of those is a correctness problem at this size.
215+
216+
Three consequences of the shape, since they are what a reader of the code will meet:
217+
a run's header is one immutable JSON file under `fleets/<fleet>/runs/`, written with
218+
`wx` exactly the way a member's record is; its steps are `run.start`, `run.step` and
219+
`run.end` lines on the fleet's existing `ledger.jsonl`, so `moshcode fleet log` reads
220+
them with no second tool; and a heartbeat is deliberately NOT in the ledger, because
221+
it is the one fact here that is worthless the moment it is superseded. A beat is a
222+
small mutable file under `fleets/<fleet>/beats/` that is rewritten in place and deleted
223+
when the run ends. Appending every beat would grow the ledger without bound to hold a
224+
value only its last line ever answers.
172225
- Open question: `.agents/skills` is a convention, not a standard. Worth checking whether
173226
anything beyond swamp reads it before treating it as a destination.

‎src/cli-schema.mjs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1597,6 +1597,31 @@ export const FLEET_VERBS = [
15971597
["--json", "one JSON object per line, as the ledger holds them", ""],
15981598
],
15991599
note: "who did what, in order: by is sysop or a member id; what each member spent, how each ended, what was refused and why." },
1600+
// PRD 0019 R2 and R3. Two verbs, additive to the spec's five: one an engine's
1601+
// hook calls on every turn, one a person runs when the record has grown.
1602+
{ name: "beat", description: "record that a run is alive, and what it is doing (PRD 0019 R2)",
1603+
synopsis: [["moshcode fleet beat <run> [--state working|blocked|done|idle] [--session <name>] [--ttl 10m] [--json]", ""]],
1604+
flags: [
1605+
["--state <state>", "what the run is doing. a beat with a state is reported as known rather than inferred", "liveness only"],
1606+
["--kind <kind>", "for a blocked state: permission, question or menu", ""],
1607+
["--fleet <id>", "the fleet the run belongs to", "$MOSHCODE_RUN_FLEET, else this process's fleet"],
1608+
["--session <name>", "the herd session the run drives", "$MOSHCODE_HERD_NAME"],
1609+
["--ttl <dur>", "how long the beat is worth believing, capped at 10m", "10m"],
1610+
["--json", "the beat as written", ""],
1611+
],
1612+
note: "the verb an engine's lifecycle hook calls. a run that beats is shown on the roster as known; a session with no run is inferred "
1613+
+ "from its screen and prints a trailing ?. heartbeats exist only for runs moshcode started, so both tiers are permanent." },
1614+
{ name: "gc", description: "prune ended runs, newest first per working directory (PRD 0019 retention)",
1615+
synopsis: [["moshcode fleet gc [fleet] [--keep 20] [--older-than 30d] [--dry-run] [--json]", ""]],
1616+
flags: [
1617+
["--keep <n>", "ended runs to keep per working directory", "20"],
1618+
["--older-than <dur>", "drop any ended run older than this, however few there are", "30d"],
1619+
["--dry-run", "say what would go and remove nothing", ""],
1620+
["--json", "the report: what was removed, why, and what was kept", ""],
1621+
],
1622+
note: "an immutable run record grows without bound, so this is how it is allowed to be immutable. a run with no run.end is never a "
1623+
+ "candidate at any age, only this host's ledger is rewritten, and only run.* lines are dropped: member.start, swarm.end and "
1624+
+ "ceiling.refuse are the record of what agents were allowed to do and no retention policy removes those." },
16001625
];
16011626

16021627
// The business layer's verbs. Flatter than the herd's on purpose: these are

‎src/fleet-cli.mjs‎

Lines changed: 120 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@ import fs from "node:fs";
1515
import path from "node:path";
1616

1717
import { EXIT, herdKill, roster as herdRoster } from "./herd-cli.mjs";
18+
import { STATES } from "./herd-state.mjs";
1819
import { herdDir, slugifyName } from "./herd.mjs";
1920
import * as fleet from "./openfleet.mjs";
21+
import * as runs from "./run-record.mjs";
2022
import { acid, amber, ash, bone, dim, err, info, ok, warn } from "./ui.mjs";
2123

2224
/** The exit a refused open, cap or stop returns. logicsrc fleet uses the same number. */
@@ -28,6 +30,8 @@ const USAGE = {
2830
tree: "usage: moshcode fleet tree [fleet] [--json]",
2931
stop: "usage: moshcode fleet stop <member|swarm> | --fleet <fleet> [--json]",
3032
log: "usage: moshcode fleet log [fleet] [--since 1h] [--member <id>] [--swarm <id>] [--json]",
33+
beat: "usage: moshcode fleet beat <run> [--state working|blocked|done|idle] [--fleet <id>] [--session <name>] [--ttl 10m] [--json]",
34+
gc: "usage: moshcode fleet gc [fleet] [--keep 20] [--older-than 30d] [--dry-run] [--json]",
3135
};
3236

3337
/* ----------------------------------------------------------------- parsing */
@@ -526,14 +530,19 @@ function describeLine(l) {
526530
case "member.end": return `${l.member} · ${l.state}${l.total ? ` · ${l.total}` : ""}${l.summary ? ` · ${clip(String(l.summary).replace(/\s+/g, " "), 80)}` : ""}`;
527531
case "swarm.end": return `${l.swarm} · ${l.state}${l.verdict ? ` · ${l.verdict.length} verdict${l.verdict.length === 1 ? "" : "s"}` : ""}${l.summary ? ` · ${clip(String(l.summary).replace(/\s+/g, " "), 80)}` : ""}`;
528532
case "ceiling.refuse": return `${l.member ? `${l.member} · ` : ""}${l.action} refused on ${l.key}: wanted ${JSON.stringify(l.wanted)}, allowed ${JSON.stringify(l.allowed)}`;
533+
// The run record rides the same ledger (PRD 0019 R3), so `fleet log`
534+
// reads it without a second tool and without being taught a second format.
535+
case "run.start": return `${l.run} · ${l.engine || "?"}${l.model ? ` ${l.model}` : ""}${l.session ? ` (${l.session})` : ""}${l.task ? ` · "${clip(l.task, 50)}"` : ""}`;
536+
case "run.step": return `${l.run} · ${l.kind}${l.name ? ` ${l.name}` : ""}${l.summary ? ` · ${clip(String(l.summary).replace(/\s+/g, " "), 80)}` : ""}`;
537+
case "run.end": return `${l.run} · ${l.state}${l.summary ? ` · ${clip(String(l.summary).replace(/\s+/g, " "), 80)}` : ""}`;
529538
default: return "";
530539
}
531540
}
532541

533542
const paintEvent = (event) => {
534543
if (event === "ceiling.refuse") return amber(event);
535544
if (event.endsWith(".end")) return ash(event);
536-
if (event === "swarm.spawn" || event === "member.start") return acid(event);
545+
if (event === "swarm.spawn" || event === "member.start" || event === "run.start") return acid(event);
537546
return bone(event);
538547
};
539548

@@ -577,9 +586,118 @@ async function fleetLog(argv, o) {
577586
return EXIT.matched;
578587
}
579588

589+
/* ------------------------------------------------------- beat and retention */
590+
591+
/**
592+
* `moshcode fleet beat <run>`: this run is alive, and here is what it is doing.
593+
*
594+
* The verb an engine's lifecycle hook calls (PRD 0019 R2). It is deliberately
595+
* the cheapest thing in this file: one file written, no ledger read, no tree
596+
* folded, because it runs on every turn of every session in the herd and a
597+
* beat that costs a tenth of a second is a beat somebody turns off.
598+
*
599+
* Unlike `open` and `cap` this is not the sysop's alone. A member beating for
600+
* its own run is the normal case and the reason the verb exists.
601+
*/
602+
export function fleetBeat(argv, { write = console.log, env = process.env, now = () => Date.now() } = {}) {
603+
const { flags, positional, errors } = parseArgs(argv, {
604+
valued: ["state", "kind", "fleet", "session", "member", "ttl"], flags: ["json"],
605+
});
606+
for (const e of errors) write(err(e));
607+
if (errors.length) { write(err(USAGE.beat)); return EXIT.usage; }
608+
609+
const run = positional[0] || env.MOSHCODE_RUN || null;
610+
if (!run) { write(err(USAGE.beat)); return EXIT.usage; }
611+
612+
// The fleet the run belongs to, in the order a caller can actually supply
613+
// it: the flag, the variable moshcode set beside MOSHCODE_RUN when it
614+
// started the session, then the fleet this process sits in.
615+
const name = flags.fleet || env.MOSHCODE_RUN_FLEET || env.OPENFLEET_FLEET
616+
|| fleet.currentFleet(env) || fleet.implicitFleet(env);
617+
618+
if (flags.state && !STATES.includes(flags.state)) {
619+
write(err(`--state is one of ${STATES.join(", ")}`));
620+
return EXIT.usage;
621+
}
622+
let ttl;
623+
if (flags.ttl !== undefined) {
624+
ttl = parseDurationMs(flags.ttl);
625+
if (ttl === null) { write(err("--ttl is a duration like 10m")); return EXIT.usage; }
626+
}
627+
628+
const written = runs.beat(name, run, {
629+
state: flags.state, kind: flags.kind, ttl,
630+
session: flags.session || env.MOSHCODE_HERD_NAME, member: flags.member || env.OPENFLEET_MEMBER,
631+
}, { env, now: now() });
632+
633+
if (!written.ok) {
634+
write(err(String(written.error?.message || written.error)));
635+
return EXIT.infra;
636+
}
637+
if (flags.json) { write(JSON.stringify(written.beat, null, 2)); return EXIT.matched; }
638+
write(ok(`${bone(run)} beat${flags.state ? ` · ${flags.state}` : ""} · fleet ${name}`));
639+
return EXIT.matched;
640+
}
641+
642+
/**
643+
* `moshcode fleet gc`: the retention the run record needs to be allowed to be
644+
* immutable (PRD 0019, Risks).
645+
*
646+
* An immutable record grows without bound, which is a reason not to ship one
647+
* unless the pruning ships with it. omp's shape: keep the newest N runs per
648+
* working directory rather than N overall, so one busy checkout cannot evict
649+
* every other one, plus an age cap for the directories nobody has touched in a
650+
* month. An unfinished run is never a candidate at any age.
651+
*
652+
* `--dry-run` first is the polite default to recommend, not to impose: this
653+
* deletes history somebody may want, and a verb that is hard to preview is a
654+
* verb people run blind.
655+
*/
656+
export function fleetGc(argv, { write = console.log, env = process.env, now = () => Date.now() } = {}) {
657+
const { flags, positional, errors } = parseArgs(argv, {
658+
valued: ["keep", "older-than", "fleet"], flags: ["json", "dry-run"],
659+
});
660+
for (const e of errors) write(err(e));
661+
if (errors.length) { write(err(USAGE.gc)); return EXIT.usage; }
662+
663+
let keep = runs.GC_KEEP_PER_DIR;
664+
if (flags.keep !== undefined) {
665+
keep = Number(flags.keep);
666+
if (!Number.isInteger(keep) || keep < 0) { write(err("--keep is a whole number")); return EXIT.usage; }
667+
}
668+
let maxAgeMs = runs.GC_MAX_AGE_MS;
669+
if (flags["older-than"] !== undefined) {
670+
maxAgeMs = parseDurationMs(flags["older-than"]);
671+
if (maxAgeMs === null) { write(err("--older-than is a duration like 30d")); return EXIT.usage; }
672+
}
673+
674+
const only = flags.fleet || positional[0] || null;
675+
const report = runs.gcRuns({ env, now: now(), keep, maxAgeMs, fleet: only, dryRun: Boolean(flags["dry-run"]) });
676+
677+
if (flags.json) { write(JSON.stringify(report, null, 2)); return EXIT.matched; }
678+
const verb = report.dryRun ? "would remove" : "removed";
679+
if (!report.removed.length) {
680+
write(info(`nothing to remove. ${report.kept} run${report.kept === 1 ? "" : "s"} kept, ${keep} per directory.`));
681+
return EXIT.matched;
682+
}
683+
for (const r of report.removed) {
684+
write(`${dim(r.ended)} ${bone(r.run)} ${ash(tildeHome(r.cwd))} ${dim(r.why === "age" ? "older than the age cap" : "beyond the per-directory count")}`);
685+
}
686+
write(ok(`${verb} ${report.removed.length} run${report.removed.length === 1 ? "" : "s"} and ${report.lines} ledger line${report.lines === 1 ? "" : "s"}. ${report.kept} kept.`));
687+
return EXIT.matched;
688+
}
689+
690+
const tildeHome = (p) => {
691+
const h = process.env.HOME || "";
692+
return h && String(p).startsWith(h) ? `~${String(p).slice(h.length)}` : String(p ?? "");
693+
};
694+
580695
/* --------------------------------------------------------------- dispatch */
581696

582-
const VERBS = { open: fleetOpen, cap: fleetCap, tree: fleetTree, stop: fleetStop, log: fleetLog };
697+
const VERBS = {
698+
open: fleetOpen, cap: fleetCap, tree: fleetTree, stop: fleetStop, log: fleetLog,
699+
beat: fleetBeat, gc: fleetGc,
700+
};
583701

584702
export async function fleetCommand(argv = [], { write = console.log, ...deps } = {}) {
585703
const o = {

0 commit comments

Comments
 (0)