fix(daemon): pause dispatch on low disk and carry workerSandbox to the runner (AGT-4785)

CHANGELOG.md

@@ -18,6 +18,8 @@
 
 ### Added
 
+- **The daemon stops dispatching when the disk is nearly full (AGT-4785).** On 2026-10-20 the host ran out of disk: tester and worker stages failed with `ENOSPC` copying git packs into `$TMPDIR/.openswarm-verify-head-*`, most of the night's runs ended `infra_error`, and every new heartbeat claimed more work that could only fail the same way. `src/support/hostHealth.ts` measures free bytes (`statfs`, `bavail` — the root-reserved blocks are not ours) on every volume the daemon writes to (each allowed project, the temp dir) and the heartbeat pauses dispatch when the scarcest is under `autonomous.minFreeDiskGiB` (default 10; 0 switches the check off). The check sits after the worktree sweep, which is what frees space, so a full disk stops new work and never the cleanup that would end the condition. One notice on the way down and one on recovery — not one per heartbeat. Reads fail open: a path that cannot be measured is skipped, and with nothing measurable dispatch continues. `isHostResourceError` recognises `ENOSPC` by code or by the text git and node print. **Not done:** an explicit dispatch (`/api/work`, `exec --issue`) is not held by this gate, and a run that already failed with `ENOSPC` is still classified as a plain `infra_error`.
+
 - **`projectAgents[].setup` — dependency install commands that run in the worktree before the worker (AGT-4798).** They run with the network open but inside the sandbox and without credentials, in order, stopping at the first failure; a failed setup ends the run as `infra_error` before any model call. Most projects need none.
 
 - **`openswarm exec --issue INT-123` deploys the worker pipeline for one Linear issue asynchronously and hands the run to an external orchestrator.** It resolves the issue, checks the repo's Linear project mapping and the deployable state, records an **exec hold**, and posts to the daemon's `/api/work` (falling back to a detached `work` child only when the daemon is unreachable — never on an HTTP refusal, so a 403 from `autonomous.allowedProjects` cannot be routed around), then prints `{runId, identifier, transport, deployed}` and returns. `exec status <runId>`, `exec await <runId> --timeout <s>` and `exec answer <runId> [--correlation hq-…] [--text …]` are the observation half: `await` blocks until the run elevates or finishes and prints the structured elevation JSON (state, park reason, branch/worktree/PR, pending questions with their class, and the exact next command), and `answer` writes the reply to the coordination board as `actorRole: 'orchestrator'` — the same delegated authority OpenSwarm's own supervisor answers with — resumes the ledger row through the exact-correlation park contract first, re-marks the hold, and redeploys. All four take `--json`. Reads use a read-only handle (`{ readonly: true }`, the `ledgerOverviewTool` pattern) so `status`/`await` never create a database, migrate a schema, or take a write lock on a live daemon's file.
@@ -32,6 +34,7 @@
 
 ### Fixed
 
+- **`autonomous.workerSandbox` never reached the runner, so the worker's bash ran unfenced while the boot log said "Worker sandbox: on" (found while wiring AGT-4785).** The schema defaults it to `on` and the boot log reads it from the config object, but `core/service.ts` builds the runner's config by hand-picked mapping and did not carry the field; the runner then saw `undefined`, and the bash tool fences only on an explicit `on` (`tools.ts`, `execOptions?.sandbox === 'on'`; `pairPipelineTypes.ts`: "Unset = off"). The setup step defaults an unset value to `on`, so it was fenced while the worker's own shell was not. Every control that rides on the sandbox wrapper for the worker's shell — the loopback-only network (AGT-4798) and the write fence — was inactive in a daemon started through `service.ts`; the credential-free environment (AGT-4783) does not depend on it and was in effect. The same class as AGT-4122 and AGT-4404. The field is now mapped, and `service.test.ts` asserts it (removing the mapping makes that test fail). Not checked: whether the CLI entry points that build their own runner config carry the field.
 - **The test suite no longer writes into the operator's live `~/.openswarm/task-state.json`.** `vitest.setup.ts` redirected a dozen home-dir state paths per worker but missed `OPENSWARM_TASK_STATE_FILE`, so suites that built a real task-state store fell back to the default path and appended fixture rows (`issue-uuid`/"Serial task", `INT-1`) to the live file. It is now redirected per worker like the rest.
 
 ## 0.24.4 — 2026-09-29

config.example.yaml

@@ -272,6 +272,12 @@ autonomous:
   # that need the network go in projectAgents[].setup below.
   # workerNetwork: loopback
 
+  # Free-space floor, in GiB, for every volume the daemon writes to (each
+  # allowed project, the temp dir). Below it the heartbeat stops fetching and
+  # claiming work — worktree cleanup keeps running — and resumes once space is
+  # freed; one notice each way. 0 switches the check off. (AGT-4785)
+  # minFreeDiskGiB: 10
+
   # Delegated CLI adapters (claude, codex, cursor) own their tool loop, so
   # OpenSwarm's publication fence, protected files, role tool limits and the
   # credential-free sandboxed shell do not reach them (claude even runs with

src/automation/autonomousRunner.hostHealth.test.ts

@@ -0,0 +1,104 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { TaskItem } from '../orchestration/decisionEngine.js';
+import type { ITaskSource } from './taskSource.js';
+import type { DurableRunCoordinator } from './durableRunCoordinator.js';
+
+// AGT-4785: on 2026-10-20 the host ran out of disk and the daemon kept claiming work that could
+// only fail with ENOSPC. HostDiskGate is unit-tested in support/hostHealth.test.ts; this proves
+// the heartbeat consults it with the configured floor, before anything is fetched.
+
+vi.mock('../core/providerOverride.js', () => ({ writeProviderOverride: vi.fn() }));
+vi.mock('../agents/stageModelResolver.js', () => ({ resolveAdapterDefaultModel: vi.fn(async () => 'model') }));
+vi.mock('../memory/repoKnowledge.js', () => ({
+  recordTaskOutcome: vi.fn(async () => {}),
+  promoteStagedMemories: vi.fn(async () => 0),
+}));
+vi.mock('../linear/projectUpdater.js', () => ({ updateProjectAfterTask: vi.fn(async () => {}) }));
+
+type InternalRunner = {
+  heartbeatParallel: ReturnType<typeof vi.fn>;
+  refreshKnowledgeGraphs(): void;
+  durableRuns: DurableRunCoordinator;
+};
+
+// No real volume has this much free space, so a floor this high is always "low".
+const UNREACHABLE_FLOOR_GIB = 1e9;
+
+describe('heartbeat holds dispatch when the disk is nearly full (AGT-4785)', () => {
+  let root: string;
+
+  beforeEach(() => {
+    vi.useFakeTimers();
+    vi.setSystemTime(new Date('2026-07-21T16:00:00Z')); // 01:00 KST: work window allowed
+    root = mkdtempSync(join(tmpdir(), 'openswarm-runner-hosthealth-'));
+    vi.stubEnv('OPENSWARM_TASK_STATE_FILE', join(root, 'task-state.json'));
+    vi.stubEnv('OPENSWARM_RUNNER_TASK_STATE_FILE', join(root, 'runner-state.json'));
+    vi.stubEnv('OPENSWARM_RUNNER_REJECTION_STATE_FILE', join(root, 'rejections.json'));
+    vi.stubEnv('OPENSWARM_RUNNER_PIPELINE_HISTORY_FILE', join(root, 'history.json'));
+    vi.stubEnv('OPENSWARM_RUNNER_DECOMPOSITION_STATE_FILE', join(root, 'decomposition.json'));
+  });
+
+  afterEach(() => {
+    vi.useRealTimers();
+    vi.unstubAllEnvs();
+    vi.restoreAllMocks();
+    rmSync(root, { recursive: true, force: true });
+    vi.resetModules();
+  });
+
+  async function setup(minFreeDiskGiB: number | undefined) {
+    const [{ AutonomousRunner }, execution] = await Promise.all([
+      import('./autonomousRunner.js'),
+      import('./runnerExecution.js'),
+    ]);
+    const fetchTasks = vi.fn(async () => [] as TaskItem[]);
+    const source = {
+      kind: 'linear', fetchTasks,
+      lookupIssueState: vi.fn(async () => ({ ok: true as const, issue: { state: 'Todo', stateType: 'unstarted' } })),
+      updateState: vi.fn(async () => true), addComment: vi.fn(async () => {}),
+      createTask: vi.fn(), createSubIssue: vi.fn(), logPairStart: vi.fn(),
+      logPairComplete: vi.fn(), logBlocked: vi.fn(), logStuck: vi.fn(), unstick: vi.fn(),
+      logHalt: vi.fn(), markAsDecomposed: vi.fn(),
+    } as unknown as ITaskSource;
+    execution.setTaskSource(source);
+    const runner = new AutonomousRunner({
+      linearTeamId: 'team', allowedProjects: [root], heartbeatSchedule: '0 * * * *',
+      autoExecute: true, dryRun: true, pairMode: true, maxConcurrentTasks: 4,
+      automationLedgerMode: 'primary', automationDbPath: join(root, 'hosthealth.db'),
+      ...(minFreeDiskGiB === undefined ? {} : { minFreeDiskGiB }),
+    });
+    const internal = runner as unknown as InternalRunner;
+    internal.refreshKnowledgeGraphs = vi.fn();
+    internal.heartbeatParallel = vi.fn(async () => {});
+    return { runner, internal, fetchTasks };
+  }
+
+  it('fetches and claims nothing while free space is under the configured floor', async () => {
+    const { runner, internal, fetchTasks } = await setup(UNREACHABLE_FLOOR_GIB);
+    await runner.heartbeat();
+    internal.durableRuns.close();
+    expect(fetchTasks).not.toHaveBeenCalled();
+    expect(internal.heartbeatParallel).not.toHaveBeenCalled();
+  }, 60_000);
+
+  it('says so once per change of state, not once per heartbeat', async () => {
+    const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+    const { runner, internal } = await setup(UNREACHABLE_FLOOR_GIB);
+    await runner.heartbeat();
+    await runner.heartbeat();
+    await runner.heartbeat();
+    internal.durableRuns.close();
+    const pauses = warn.mock.calls.filter((c) => String(c[0]).includes('Dispatch paused'));
+    expect(pauses).toHaveLength(1);
+  }, 60_000);
+
+  it('a floor of 0 switches the check off and the heartbeat proceeds to fetch', async () => {
+    const { runner, internal, fetchTasks } = await setup(0);
+    await runner.heartbeat();
+    internal.durableRuns.close();
+    expect(fetchTasks).toHaveBeenCalled();
+  }, 60_000);
+});

src/automation/autonomousRunner.ts

@@ -70,6 +70,7 @@ import { SANDBOX_OUTCOME_UNKNOWN_PARK_REASON } from '../sandboxExecutor/protocol
 import { INFRA_CIRCUIT_PARK_REASON } from './infraFailureCircuit.js';
 import { listHeldIssueIds } from './execHold.js';
 import { inputFingerprint, mayRedispatch } from './dispatchFingerprint.js';
+import { HostDiskGate, dispatchWritePaths } from '../support/hostHealth.js';
 import { VERIFIED_WORKER_BLOCKER_PARK_REASON } from '../agents/pairPipelineTypes.js';
 import { decideExplicitReadmission, OPERATOR_QUESTION_PARK_MARKER } from './explicitDispatchReadmission.js';
 import { broadcastEvent, type SwarmStats } from '../core/eventHub.js';
@@ -533,6 +534,11 @@ export class AutonomousRunner {
   // Rate-limit hold: epoch ms until which all task execution is paused.
   // Set when any adapter returns a 429 / usage_limit_reached response (INT-1906).
   private rateLimitUntil = 0;
+  // Disk-space hold: new claims wait while the volumes we write to are nearly full (AGT-4785).
+  private readonly diskGate = new HostDiskGate(
+    () => dispatchWritePaths(this.config.allowedProjects.map((p) => normalizeProjectPath(p))),
+    () => (this.config.minFreeDiskGiB ?? 10) * 1024 ** 3,
+  );
 
   // Issues whose Linear project can't be mapped to a local repo path. Recorded on
   // the first resolve failure so they aren't re-picked every heartbeat (which
@@ -2838,6 +2844,19 @@ export class AutonomousRunner {
       }
       this.syslog('✓ Time window: allowed');
 
+      // 1.1 Disk-space hold — after the worktree sweep above, which is what frees
+      // space, and before anything is fetched or claimed. One notice per change of
+      // state, not one per heartbeat. (AGT-4785)
+      const disk = this.diskGate.evaluate();
+      if (disk.notice) {
+        console.warn(`[AutonomousRunner] ${disk.notice}`);
+        await reportToDiscord(`${disk.pause ? '⚠️' : '✅'} ${disk.notice}`);
+      }
+      if (disk.pause) {
+        this.syslog(`⏸ Disk space hold: ${disk.line}`);
+        return;
+      }
+
       // 1.2 Rate-limit hold — skip the heartbeat while a 429 pause is still active.
       // Cleared implicitly once the clock passes rateLimitUntil. (INT-1906)
       if (Date.now() < this.rateLimitUntil) {

src/automation/runnerTypes.ts

@@ -86,6 +86,8 @@ export interface AutonomousConfig {
   adapterRouting?: { primary?: import('../adapters/types.js').AdapterName; fallbacks?: Array<'cc-router' | 'cursor' | 'codex' | 'codex-responses'>; allowReasons?: Array<'quota' | 'infra' | 'capability'> };
   /** OS fence for the worker's bash tool (AGT-4387). */
   workerSandbox?: 'on' | 'off';
+  /** Free-space floor (GiB) below which dispatch pauses; 0 switches the check off (AGT-4785). */
+  minFreeDiskGiB?: number;
   /** Periodic read-only repository review jobs. */
   periodicReviews?: Array<{ profile: 'permissions' | 'hygiene' | 'security' | 'review'; schedule: string; adapter?: 'codex' | 'cc-router' | 'cursor' }>;
   /** Explicit high-capability project supervisor. */

src/core/config.test.ts

@@ -371,6 +371,22 @@ agents:
       expect(agents?.[1].goal).toBeUndefined();
     });
 
+    it('carries minFreeDiskGiB to the runtime config and defaults the floor to 10 GiB (AGT-4785)', () => {
+      vi.mocked(existsSync).mockReturnValue(true);
+      const load = (autonomous: Record<string, unknown>) => {
+        vi.mocked(readFileSync).mockReturnValue(JSON.stringify({
+          language: 'en',
+          linear: { apiKey: 'k', teamId: 't' },
+          agents: [{ name: 'main', projectPath: '/p', enabled: true, paused: false }],
+          autonomous: { enabled: true, ...autonomous },
+        }));
+        return loadConfig('/tmp/config.json').autonomous;
+      };
+      expect(load({})?.minFreeDiskGiB).toBe(10);
+      expect(load({ minFreeDiskGiB: 25 })?.minFreeDiskGiB).toBe(25);
+      expect(load({ minFreeDiskGiB: 0 })?.minFreeDiskGiB).toBe(0);
+    });
+
     it('carries workerNetwork and a project setup list to the runtime config (AGT-4798)', () => {
       vi.mocked(existsSync).mockReturnValue(true);
       vi.mocked(readFileSync).mockReturnValue(JSON.stringify({

src/core/config.ts

@@ -481,6 +481,8 @@ const AutonomousConfigSchema = z.object({
   workerSandbox: z.enum(['on', 'off']).default('on'),
   /** Accept delegated CLI adapters (claude/codex/cursor) for unattended work despite the missing fences (AGT-4444) */
   allowDelegatedAdapters: z.boolean().default(false),
+  /** Free disk space (GiB) below which dispatch pauses; 0 switches the check off (AGT-4785) */
+  minFreeDiskGiB: z.number().min(0).default(10),
   /** Network for the worker's shell: loopback only (default) or open (AGT-4798) */
   workerNetwork: z.enum(['loopback', 'open']).default('loopback'),
   periodicReviews: z.array(z.object({
@@ -890,6 +892,7 @@ function transformConfig(raw: RawConfig): SwarmConfig {
       adapterRouting: raw.autonomous.adapterRouting,
       workerSandbox: raw.autonomous.workerSandbox,
       workerNetwork: raw.autonomous.workerNetwork,
+      minFreeDiskGiB: raw.autonomous.minFreeDiskGiB,
       allowDelegatedAdapters: raw.autonomous.allowDelegatedAdapters,
       periodicReviews: raw.autonomous.periodicReviews,
       orchestrator: raw.autonomous.orchestrator,

src/core/service.test.ts

@@ -299,6 +299,19 @@ describe('service', () => {
     );
   });
 
+  it('forwards workerSandbox and minFreeDiskGiB to the runner — the mapping dropped workerSandbox, so the fence was off while the log said on (AGT-4785)', async () => {
+    const { startAutonomous } = await import('../automation/autonomousRunner.js');
+
+    await startService({
+      ...mockAutonomousConfig,
+      autonomous: { ...mockAutonomousConfig.autonomous, workerSandbox: 'on', minFreeDiskGiB: 7 },
+    } as SwarmConfig);
+
+    expect(vi.mocked(startAutonomous)).toHaveBeenCalledWith(
+      expect.objectContaining({ workerSandbox: 'on', minFreeDiskGiB: 7 }),
+    );
+  });
+
   it('reapplies a persisted provider override on boot when it differs from config', async () => {
     const { readProviderOverride } = await import('./providerOverride.js');
     const { setDefaultAdapter } = await import('../adapters/index.js');

src/core/service.ts

@@ -394,6 +394,12 @@ async function startServiceLocked(config: SwarmConfig): Promise<void> {
       coordinationBoardIssueId: config.autonomous.coordinationBoardIssueId,
       mcpPolicies: config.autonomous.mcpPolicies,
       adapterRouting: config.autonomous.adapterRouting,
+      // Same trap as AGT-4122/AGT-4404: the schema, the boot log below and the
+      // pipeline all read workerSandbox, but this hand-picked mapping did not
+      // pass it, so the runner saw undefined and the bash tool (which fences only
+      // on an explicit 'on') ran unfenced while the log said "sandbox: on".
+      workerSandbox: config.autonomous.workerSandbox,
+      minFreeDiskGiB: config.autonomous.minFreeDiskGiB, // AGT-4785
       periodicReviews: config.autonomous.periodicReviews,
       orchestrator: config.autonomous.orchestrator,
       orchestratorSchedule: config.autonomous.orchestratorSchedule,

src/core/types.ts

@@ -717,6 +717,12 @@ export type AutonomousStartupConfig = {
    * Dependency installs that need the network go in projectAgents[].setup.
    */
   workerNetwork?: 'loopback' | 'open';
+  /**
+   * Free disk space, in GiB, below which the heartbeat stops claiming new work
+   * (AGT-4785). Worktree cleanup keeps running. 0 switches the check off.
+   * Default 10.
+   */
+  minFreeDiskGiB?: number;
   /**
    * Run unattended work on delegated CLI adapters (claude/codex/cursor) even
    * though OpenSwarm's publication fence, protected files and credential-free