Skip to content

Commit 39b2dd1

Browse files
committed
fix(workflow): end a run whose running node the graph edit drops
saveGraph rebuilt the node map against the new graph but left the run running with no node in flight, so under maxConcurrent the slot was held forever: the probe read status running, nodeStatus queued-only and no way to advance. A dropped running node now ends its run the way Stop/Reset/ dispose do: the node is failed, the run stays failed with that node in its map, its node Bots are released and the queue drains (#1549).
1 parent bf5e878 commit 39b2dd1

4 files changed

Lines changed: 145 additions & 18 deletions

File tree

‎changelog.d/1549.fixed.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Deleting a running node from the workflow canvas no longer strands its run: the edit ends that run the way Stop does — the node is failed, the run is archived `failed`, its node Bots are released and a queued task takes the lane. Before, the run stayed `running` with a queued-only node map and no timer, so under `maxConcurrent: 1` nothing could ever start behind it. ([#1549](https://github.com/ThinkFlowLab/FlyBot/issues/1549))

‎docs/subsystems/workflow.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher,
4646

4747
1. Friends' Bots are not nodes. A `remote` Bot is refused by create, define and the matcher.
4848
2. The canvas and the run share one graph: nodes, edges and positions. A drag commits `workflow/graph`, but only while **Edit** is on — a factory preset is copied first (#1378). Status highlighting reads that same graph. Replacing edges does not recompute who is already running. The Settings canvas paints that graph on the same 960px design plane, scaled to the width the settings page gives it, so the canvas is not a second scroll container inside the page's own scroll; its pointer deltas are divided by the scale, and its wires keep a stronger stroke than a plain divider so an edge can be followed at that size (#1483).
49-
3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538):** When a node completes with `needs-human` or `already-claimed` detected in its outputs or reply text (see `checkEarlyEnd` in Key files), the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to or spawn successor nodes. This allows Triage, Claim, or any step to terminate the pipeline cleanly when human intervention is required or the issue is already owned by another agent. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it. A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). Every wait is bounded (#1530). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running step also holds a deadline — `STEP_DEADLINE_MS`, ten minutes, one constant in `service.js`; when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat says the step timed out, and `warn` names it once. The deadline is cleared as soon as the step moves, and by Pause, Stop, Reset, delete, `reap` and `dispose`; a paused run holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply that says `could not answer your message` or `model call failed` fails that node and does not start its successor. A 429 or rate-limit mention in any other reply does not.
49+
3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538):** When a node completes with `needs-human` or `already-claimed` detected in its outputs or reply text (see `checkEarlyEnd` in Key files), the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to or spawn successor nodes. This allows Triage, Claim, or any step to terminate the pipeline cleanly when human intervention is required or the issue is already owned by another agent. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it. A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). A graph edit that drops a node that is still running ends that run the way Stop does: the node is failed, the run is archived `failed`, its node Bots are released, and the queue drains (#1549). Every wait is bounded (#1530). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running step also holds a deadline — `STEP_DEADLINE_MS`, ten minutes, one constant in `service.js`; when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat says the step timed out, and `warn` names it once. The deadline is cleared as soon as the step moves, and by Pause, Stop, Reset, delete, `reap` and `dispose`; a paused run holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply that says `could not answer your message` or `model call failed` fails that node and does not start its successor. A 429 or rate-limit mention in any other reply does not.
5050
4. Each workflow keeps a FIFO queue and up to `maxConcurrent` live runs (default 2, an integer from 1 to 8 on the record, via `workflow/concurrency`, or in Settings → Workflow graphs). A task that does not fit is enqueued. When a run finishes, every workflow with a free slot is drained, oldest enqueue first. A new assign on a workflow that already has a queue waits behind it. A task does not replace a running run or release that run's Bots. Migrating an old store keeps only runs that were still `running` with ephemeral Bot ids; other legacy scripted rows are dropped from active state (finished tasks may remain in lane history). History records a finished or failed live run. Lowering the cap does not stop a run that has already started. Assigning `{ botId }` claims the card for that local Bot and does not enqueue. A remote Bot is refused. `activeCount` is `runningOf(lane)` — live runs with `live === true` and `status === 'running'`. The stage aggregate chip and the Runs tab (运行记录) must agree: every such run appears as its own running row and progress card, and the chip count does not exceed that list.
5151
5. A node declares up to eight inputs and eight outputs. A required input is an output of the same name on an ancestor, or `task`, `title`, or `brief` on a root. A start with no provider is refused, and Bots spawned for that attempt are released. On a Bot-backed run a successor stays queued until its required inputs are present. If every forward predecessor is done, failed or skipped and an input is still missing, that node fails and the run stops. The dispatch text carries `Input <name>: <value>`. The Bot reports `OUTPUT <name>: <value>`; a single declared output may be the reply body. Nodes start in edge order.
5252
6. Each node is a real ephemeral Bot for that run, then it is removed. A permanent sidebar Bot (`botId`) is optional and is not required for the step to run. **File and shell rights are the same as a normal sidebar Bot's:** every node Bot gets the `danger-full-access` preset, and `EPHEMERAL_DENIED_TOOLS` still refuses the team-admin tools at the guard and approval layers (#1466, 29 names). **`node.tools` selects connectors only:** `spawn` passes the node's cleaned list to `fb.bot.create`, the host records it as `stepTools`, and `connector-access.js` allows `mcp__github__*` for `github` and `mcp__composio__*` for `composio` — a connector the node did not name is refused, and a node that names none reaches none. The list is also stored in `fields.tools` for the UI. A hidden run Bot cannot show an approval card, so the approval seam answers the same way the guard does. The host stores this on the run Bot as `stepTools` and `permission`.

‎plugins/workflow/src/service.js‎

Lines changed: 62 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1088,8 +1088,12 @@ export function createWorkflowService({
10881088
},
10891089
// Replace the graph the canvas and the run share. Positions and edges
10901090
// persist; a node that remains keeps its run status, a new one waits.
1091+
// A node dropped while it is running takes its run with it: the run is
1092+
// ended the way Stop/Reset/dispose end one, so nothing is left `running`
1093+
// with no node to advance it and no way to release the lane (#1549).
10911094
saveGraph(body = {}) {
1092-
return mutate(String(body.workflowId ?? ''), (item) => {
1095+
const endedRuns = []
1096+
const saved = mutate(String(body.workflowId ?? ''), (item) => {
10931097
const seated = seat(body.nodes, bots() ?? [])
10941098
const seenBots = new Set()
10951099
for (const node of seated) {
@@ -1103,34 +1107,75 @@ export function createWorkflowService({
11031107
edges: body.edges,
11041108
layers: undefined,
11051109
})
1106-
item.nodes = next.nodes
1107-
item.edges = next.edges
1108-
item.layers = next.layers
11091110
const lane = laneOf(item.id)
1110-
const run = displayRun(item.id)
1111-
if (run?.live === true) {
1112-
const ids = new Set(item.nodes.map(node => node.id))
1113-
const nodeStatus = {}
1114-
for (const id of ids) nodeStatus[id] = run.nodeStatus?.[id] ?? 'queued'
1115-
const nextRun = { ...run, nodeStatus, updatedAt: now() }
1116-
if (lane.active.some(itemRun => itemRun.id === run.id)) storeRun(item.id, nextRun)
1111+
const liveRuns = lane.active.filter(itemRun => itemRun.live === true)
1112+
if (lane.last?.live === true && !liveRuns.some(itemRun => itemRun.id === lane.last.id)) liveRuns.push(lane.last)
1113+
const ids = new Set(next.nodes.map(node => node.id))
1114+
for (const run of liveRuns) {
1115+
// endRun settles the node map against the graph it ran on, so it must
1116+
// see the old nodes; the next graph is applied after this loop.
1117+
const dropped = orderedIds(item, run.nodeStatus, 'running').filter(id => !ids.has(id))
1118+
const names = dropped.map(id => nodeById(item, id)?.name ?? id)
1119+
const ended = dropped.length > 0
1120+
const settled = ended ? endRun(item, run) : run
1121+
// A run that continues settles against the new graph, so a node this edit
1122+
// dropped leaves its map. A run this edit ended keeps its own map — the
1123+
// failed node included — the way reset() and dispose() keep theirs (#1424).
1124+
let nodeStatus = settled.nodeStatus ?? {}
1125+
if (!ended) {
1126+
nodeStatus = {}
1127+
for (const id of ids) nodeStatus[id] = settled.nodeStatus?.[id] ?? 'queued'
1128+
}
1129+
const nextRun = { ...settled, nodeStatus, updatedAt: now() }
1130+
if (ended) endedRuns.push({ run: nextRun, names })
1131+
else if (lane.active.some(itemRun => itemRun.id === run.id)) storeRun(item.id, nextRun)
11171132
else if (lane.last?.id === run.id) lane.last = nextRun
11181133
syncTimers(item, nextRun)
11191134
}
1135+
item.nodes = next.nodes
1136+
item.edges = next.edges
1137+
item.layers = next.layers
1138+
for (const { run, names } of endedRuns) {
1139+
lane.active = lane.active.filter(itemRun => itemRun.id !== run.id)
1140+
lane.last = run
1141+
lane.history = [...lane.history, { task: run.task, status: run.status, at: run.updatedAt }].slice(-20)
1142+
push(item.id, [{ kind: 'system', tone: 'bad', text: `■ 已停止 · ${summarizeTask(run.task)}(图编辑删除了正在运行的节点:${names.join('、')})` }], now())
1143+
}
1144+
})
1145+
if (saved.ok === false || endedRuns.length === 0) return saved
1146+
const releasing = endedRuns.flatMap(({ run }) => Object.values(run.ephemeral ?? {}))
1147+
for (const { run } of endedRuns) {
1148+
delete run.ephemeral
1149+
delete run.handed
1150+
}
1151+
write()
1152+
if (typeof spawn !== 'function') {
1153+
drainSync()
1154+
return present()
1155+
}
1156+
return hostSerial(async () => {
1157+
await releaseAll(releasing, release, warn)
1158+
await drainAll()
1159+
return present()
11201160
})
11211161
},
11221162
// A Bot (or the owner) writes a graph. A name that already exists is replaced.
11231163
define(body = {}) {
11241164
const existing = body.workflow ? findWorkflow(body.workflow) : undefined
11251165
if (existing) {
1166+
const finish = () => {
1167+
const name = cleanLine(body.name, 80)
1168+
if (name) {
1169+
existing.name = name
1170+
write()
1171+
}
1172+
return present()
1173+
}
11261174
const saved = this.saveGraph({ workflowId: existing.id, nodes: body.nodes, edges: body.edges ?? [] })
1175+
// saveGraph returns a promise when the edit had to end a live run.
1176+
if (typeof saved?.then === 'function') return saved.then(value => (value?.ok === false ? value : finish()))
11271177
if (saved.ok === false) return saved
1128-
const name = cleanLine(body.name, 80)
1129-
if (name) {
1130-
existing.name = name
1131-
write()
1132-
}
1133-
return present()
1178+
return finish()
11341179
}
11351180
return this.create({ name: body.name, nodes: body.nodes, edges: body.edges ?? [], description: body.description })
11361181
},

0 commit comments

Comments
 (0)