Skip to content

feat(smithy,smithy-web): dispatch stuck-queue warning (#59) - #1

Open
komoreka wants to merge 8 commits into
masterfrom
feat/dispatch-stuck-warning
Open

komoreka wants to merge 8 commits into
masterfrom
feat/dispatch-stuck-warning

Conversation

@komoreka

@komoreka komoreka commented May 4, 2026

Copy link
Copy Markdown
Owner

Internal review PR before opening upstream against stoneforge-ai/stoneforge.

Summary

Closes upstream stoneforge-ai#59. Surfaces a clear signal when the dispatch worker queue is stuck (ready unassigned tasks > 0, no available workers). Three surfaces:

  • CLI log: per-tick warning [dispatch] N task(s) ready, no available workers..., rate-limited to once per 20 ticks (configurable via DispatchDaemonConfig.stuckWarnTickInterval), re-warns immediately after the queue clears and re-stuckens.
  • HTTP: GET /api/daemon/status now includes a health field with { readyUnassignedTasks, availableWorkers, stuck, hasStuckQueue, computedAt }.
  • smithy-web: a yellow DispatchHealthBanner on the agents and workspaces pages when hasStuckQueue is true. Dismissible per page-load.

The model is intentionally simple: existing dispatch logic does not route by role at the data layer (assignTaskToWorker in dispatch-daemon.ts:2705 grabs the first unassigned task without filtering by role), so the stuck-queue concept is exclusively about workers. Directors are interactive (humans drive them); stewards run on triggers; neither pulls from the unassigned queue. An "available worker" is one that is registered, not disabled, and not terminated. At-capacity workers do not count as stuck (the queue is busy, not stuck) to avoid misleading warnings.

Validation

  • bun test src/services/dispatch-daemon.bun.test.ts covers four detection cases (no tasks, no workers, available worker, disabled worker) plus the warn-on-tick test. 1549 pass / 0 fail in smithy.
  • vitest src/server/routes/daemon.test.ts covers status payload with daemon, without daemon, and with getDispatchHealth throwing. 42 pass / 0 fail in routes suite.
  • NODE_ENV=development npx turbo run typecheck passes (16/16 packages).
  • NODE_ENV=development pnpm build passes (13/13).
  • Manual smoke pending (do this in cloudboostup-website before opening upstream PR).

Cross-PR dependency note

This branch contains a 5-line type addition (disabled?: boolean on BaseAgentMetadata) and a 10-line file (isAgentDisabled / isAgentEnabled helpers) that overlap with PR stoneforge-ai#60 against upstream. The duplication is byte-identical (verified during Task 1 review); when PR stoneforge-ai#60 merges first the rebase here is trivial.

Out of scope

Manual smoke recipe

cd /Users/adriankomorek/git/github/CloudBoostUP/cloudboostup-website
sf agent disable <some-worker-id>
sf task create --title "stuck-queue smoke test" --priority 2
kill \$(lsof -ti :3457) 2>/dev/null
sf serve smithy
# Expect: within ~10s, log line "[dispatch] 1 task(s) ready, no available workers..."
# Hard-refresh http://localhost:3457/agents — yellow banner appears.
sf agent enable <id>
# Expect: banner clears within ~5s, log line stops.
sf task close <task-id>

komoreka and others added 8 commits May 5, 2026 08:55
… detection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds DispatchHealth interface to api/types.ts, extends DaemonStatusResponse
in useDaemon.ts with an optional health field (also tightens poll to 5s),
and creates the DispatchHealthBanner component that renders an amber warning
when hasStuckQueue is true.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pages

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pper

Banner now takes an optional className applied to its outer element so
each mount site controls its own page-specific padding. The workspaces
route previously wrapped the banner in a div with px-6 pt-4 which left
16px of empty padding when the banner self-hides (queue healthy or
dismissed). The wrapper is gone; the banner mounts directly with
className="mx-6 mt-4" passed in.
Replaces the per-tick rate-limited warn with state-transition logging:
- Healthy → Stuck: single STUCK warn line with the count and a clear hint.
- Stuck → Healthy: single RESUMED info line confirming dispatch is flowing.
- No periodic reminders. A long-running stuck state does not spam the log.

Drops the stuckWarnTickInterval config option (was 20 ticks, 100s spacing
in production). Distinctive STUCK/RESUMED prefixes make the lines findable
in a chatty log stream.

Tests:
- Verifies STUCK warn fires on first stuck tick AND does not fire again
  on subsequent stuck ticks (was the periodic-spam regression risk).
- Verifies RESUMED info fires on the next healthy tick after stuck.
@komoreka
komoreka force-pushed the feat/dispatch-stuck-warning branch from f74ea8a to 7782bbe Compare May 5, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant