Skip to content

Commit 216c190

Browse files
committed
Report a turn cut off at the output limit instead of a normal finish
A `turn/end` with `reason.kind` `max-tokens` is neither a failure nor a normal finish, but the feed called it `completed`, the ended-turn view dropped it and a group member cut off before it wrote a line was recorded as a pass. The feed now carries `limited: true`, the ended view records `{ limited: true }` (with `silent` and the streak when the turn wrote nothing), the round reads `{ missed: true }` instead of `''`, and the chat says the reply was cut off rather than "could not run" (#2679).
1 parent b032d48 commit 216c190

21 files changed

Lines changed: 309 additions & 36 deletions
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
A turn that DSH ended at the model's output limit is no longer reported as a normal finish or as a member that chose silence: the activity feed marks `turn.end` with `limited: true`, the ended-turn view keeps the turn (with `silent` and the streak when it wrote nothing, and its own sentence in the chat instead of "could not run"), and a group member cut off before it answered is recorded as "no answer" instead of a pass ([#2679](https://github.com/ThinkFlowLab/FlyBot/issues/2679)).

‎core/client.js‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,8 @@ var ZH = {
122122
// #1044: a turn that died before writing anything still says so where the chat is.
123123
"Could not run: {error}": "这一轮没能跑起来:{error}",
124124
"the model call failed": "模型调用失败",
125+
// #2679: the turn did run; DSH ended it at the model's output ceiling.
126+
"The reply was cut off at the model's output limit": "回复在模型的输出上限处被截断",
125127
"{count} turns in a row": "连续 {count} 轮",
126128
"Group invite link": "群邀请链接",
127129
"Room-only: friends do not see these Bots unless you invite them.": "仅限这个房间:除非你邀请,好友看不到这些 Bot。",
@@ -4793,9 +4795,10 @@ var failureLines = (snapshot, place, t2, limit = FAILURE_LINES_SHOWN) => {
47934795
const ended = Array.isArray(snapshot?.ended) ? snapshot.ended : [];
47944796
return ended.filter((entry) => entry?.silent === true && (entry.roomId ?? entry.sessionId) === place).slice(-limit).map((entry) => ({
47954797
id: `turn-failure-${entry.id}`,
4796-
// The stored text can be a provider's JSON body cut mid-sentence; `failureSentence` says the
4797-
// same fact in words that name what to change (#1347).
4798-
text: t2("Could not run: {error}", { error: entry.failed === void 0 ? t2("the model call failed") : failureSentence(entry.failed, t2) }),
4798+
// A turn cut off at the output limit did run (#2679): it gets its own sentence, because "could not
4799+
// run" would name a failure nobody saw. The stored failure text can be a provider's JSON body cut
4800+
// mid-sentence; `failureSentence` says that same fact in words that name what to change (#1347).
4801+
text: entry.limited === true ? t2("The reply was cut off at the model's output limit") : t2("Could not run: {error}", { error: entry.failed === void 0 ? t2("the model call failed") : failureSentence(entry.failed, t2) }),
47994802
...Number(entry.streak) > 1 ? { streak: Number(entry.streak) } : {}
48004803
}));
48014804
};

‎core/src/client/i18n.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ const ZH = {
8888
// #1044: a turn that died before writing anything still says so where the chat is.
8989
'Could not run: {error}': '这一轮没能跑起来:{error}',
9090
'the model call failed': '模型调用失败',
91+
// #2679: the turn did run; DSH ended it at the model's output ceiling.
92+
'The reply was cut off at the model\'s output limit': '回复在模型的输出上限处被截断',
9193
'{count} turns in a row': '连续 {count} 轮',
9294
'Group invite link': '群邀请链接',
9395
'Room-only: friends do not see these Bots unless you invite them.': '仅限这个房间:除非你邀请,好友看不到这些 Bot。',

‎core/src/host/flybot-feed.js‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -253,8 +253,12 @@ export function installFeed(rt, fire) {
253253
closeCalls(id, entry, where, turnId, time, true)
254254
const kind = event.data?.reason?.kind
255255
const outcome = OUTCOMES.has(kind) ? kind : 'completed'
256+
// A turn DSH ended at the model's output ceiling (#2679) did end, so the three-value outcome
257+
// stays; `limited` is the fact that its text is incomplete, and a consumer that cannot read it
258+
// is no longer told the turn finished normally.
259+
const limited = kind === 'max-tokens' ? true : undefined
256260
const errorCode = outcome === 'error' ? cleanCode(event.data?.reason?.error?.code) : undefined
257-
fire('turn.end', omitUndefined({ ...base, turnId, turn: entry.turn, time, seq: seqOf(event), outcome, errorCode, model: turnModel(session, id, entry, where), tokens: entry.usage }))
261+
fire('turn.end', omitUndefined({ ...base, turnId, turn: entry.turn, time, seq: seqOf(event), outcome, errorCode, limited, model: turnModel(session, id, entry, where), tokens: entry.usage }))
258262
entry.turn = undefined
259263
entry.stepped = false
260264
entry.usage = undefined

‎core/src/host/turns.js‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -390,9 +390,12 @@ export function install(rt) {
390390
}
391391
// A turn that died before writing a line leaves the chat looking like a Bot that chose to stay
392392
// quiet (#1044), so a failure with no text is marked `silent` and carries the streak of them at the
393-
// same place: one blip and a provider that has been refusing for hours read differently.
393+
// same place: one blip and a provider that has been refusing for hours read differently. A turn DSH
394+
// cut off at the output ceiling (#2679) is kept too: it is not a failure, but its text is incomplete
395+
// and, when it wrote none, it is the same kind of quiet, which is why it joins the same streak.
394396
const noteEnded = (id, turn, info, reason, failure, calls, texts = 0, rateLimited = undefined) => {
395-
if (failure === undefined && reason?.kind !== 'completed') return
397+
const limited = failure === undefined && reason?.kind === 'max-tokens'
398+
if (failure === undefined && !limited && reason?.kind !== 'completed') return
396399
const owner = rt.groupOwners.get(id)
397400
const bot = rt.botOf(owner?.botId ?? rt.selfOf(id))
398401
if (bot === undefined || bot.remote !== undefined) return
@@ -401,7 +404,7 @@ export function install(rt) {
401404
if (owner !== undefined && rt.roomOf(owner.roomId)?.origin !== undefined) return
402405
const at = Date.now()
403406
const place = `${bot.id}|${owner?.roomId ?? ''}`
404-
const silent = failure !== undefined && texts === 0
407+
const silent = (failure !== undefined || limited) && texts === 0
405408
const lastHere = ended.findLast?.(entry => placeOfEnded(entry) === place) ?? [...ended].reverse().find(entry => placeOfEnded(entry) === place)
406409
const streak = silent ? (lastHere?.silent === true ? (lastHere.streak ?? 0) + 1 : 1) : 0
407410
ended = trimEnded([...ended, {
@@ -410,6 +413,7 @@ export function install(rt) {
410413
...(owner ? { roomId: owner.roomId } : { sessionId: id }),
411414
at,
412415
...(calls > 0 ? { calls } : {}),
416+
...(limited ? { limited: true } : {}),
413417
...(failure ? { failed: cut(failure.message || failure.code, 160).text } : {}),
414418
...(rateLimited !== undefined ? { rateLimited } : {}),
415419
...(silent ? { silent: true, streak } : {}),
@@ -729,16 +733,22 @@ export function install(rt) {
729733
}).catch(error => rt.warn('ds-bot: reply delivery failed: %s', error))
730734
}
731735
}
736+
// A turn DSH cut off at the output ceiling before it wrote a line (#2679) is not a member that
737+
// read the line and chose silence, and `''` is exactly what the round reads as a pass. The round
738+
// already tells a no-answer apart: `{ missed }`, the value a member that ran out of time leaves.
739+
const missedRoomTurn = () => reason?.kind === 'max-tokens' && replyText() === ''
732740
// A retried turn answers the same waiter or sender, so this one passes nothing on.
733741
if (!retry && info?.role === 'room') {
734742
const waiter = roomWaiters.get(info.exchangeId)
735743
roomWaiters.delete(info.exchangeId)
736744
const stopped = stoppedRoomTurns.delete(info.exchangeId)
737745
if (waiter !== undefined) {
738746
if (failure !== undefined) waiter.fail(failure)
739-
else waiter.resolve(reason?.kind === 'aborted' ? null : replyText())
747+
else if (reason?.kind === 'aborted') waiter.resolve(null)
748+
else waiter.resolve(missedRoomTurn() ? { missed: true } : replyText())
740749
} else if (!stopped && reason?.kind !== 'aborted') {
741-
// The round moved on without this member; how its turn ends still reaches the group.
750+
// The round moved on without this member; how its turn ends still reaches the group. A
751+
// cut-off turn with no text hands over `''`, which `lateRoomReply` drops like a pass.
742752
if (failure !== undefined) void rt.lateRoomFailure?.(id, info, failure)
743753
else void rt.lateRoomReply?.(id, info, replyText(), origin)
744754
}
@@ -749,6 +759,8 @@ export function install(rt) {
749759
if (failure !== undefined) void rt.lateRoomFailure?.(id, info, failure)
750760
else if (reason?.kind !== 'aborted') void rt.lateRoomReply?.(id, info, replyText(), origin)
751761
} else {
762+
// Nothing is delivered for an empty answer, cut off or not: `answer` only speaks a line
763+
// for a failure, a stop, or text the turn actually wrote.
752764
answer(info, replyText())
753765
}
754766
}

‎core/src/shared/turn-failures.js‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,12 @@ export const failureLines = (snapshot, place, t, limit = FAILURE_LINES_SHOWN) =>
1212
.slice(-limit)
1313
.map(entry => ({
1414
id: `turn-failure-${entry.id}`,
15-
// The stored text can be a provider's JSON body cut mid-sentence; `failureSentence` says the
16-
// same fact in words that name what to change (#1347).
17-
text: t('Could not run: {error}', { error: entry.failed === undefined ? t('the model call failed') : failureSentence(entry.failed, t) }),
15+
// A turn cut off at the output limit did run (#2679): it gets its own sentence, because "could not
16+
// run" would name a failure nobody saw. The stored failure text can be a provider's JSON body cut
17+
// mid-sentence; `failureSentence` says that same fact in words that name what to change (#1347).
18+
text: entry.limited === true
19+
? t('The reply was cut off at the model\'s output limit')
20+
: t('Could not run: {error}', { error: entry.failed === undefined ? t('the model call failed') : failureSentence(entry.failed, t) }),
1821
...(Number(entry.streak) > 1 ? { streak: Number(entry.streak) } : {}),
1922
}))
2023
}

‎core/test/dead-turn.test.js‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,57 @@ test('#1044: a turn that dies with no text is marked silent, once, naming the pr
9292
assert.equal(lines[0].streak, undefined)
9393
})
9494

95+
// #2679: `max-tokens` is DSH ending the turn at the model's output ceiling. It is not a failure — the
96+
// call answered — and not a normal finish, so it is recorded as `limited`, with the silence of a turn
97+
// that wrote nothing kept apart from the failure streak it used to fall out of.
98+
const limit = (emit, id, turn, texts = 0) => {
99+
emit(id, 'turn/start', { turn })
100+
for (let n = 0; n < texts; n++) emit(id, 'assistant/message', { turn, message: { content: [{ type: 'text', text: `line ${n}` }] } })
101+
emit(id, 'turn/end', { turn, reason: { kind: 'max-tokens' } })
102+
}
103+
const identity = (key, values) => (values?.error === undefined ? key : `${key.split('{error}')[0]}${values.error}`)
104+
105+
test('#2679: a turn cut off before it wrote a line is limited and silent, and says so', async (t) => {
106+
const host = await start()
107+
t.after(host.stop)
108+
limit(host.emit, 'b', 1)
109+
const [entry] = await host.ended()
110+
assert.equal(entry.limited, true)
111+
assert.equal(entry.failed, undefined, 'a cut-off turn is not a failure')
112+
assert.equal(entry.silent, true)
113+
assert.equal(entry.streak, 1)
114+
const lines = failureLines({ ended: await host.ended() }, 'b', identity)
115+
assert.equal(lines.length, 1, 'a turn cut off before it spoke is not left looking like a quiet Bot')
116+
assert.doesNotMatch(lines[0].text, /model call failed/, 'the call did not fail: it was cut off')
117+
assert.match(lines[0].text, /cut off/)
118+
})
119+
120+
test('#2679: a turn cut off mid-answer is limited but not silent, and adds no line', async (t) => {
121+
const host = await start()
122+
t.after(host.stop)
123+
limit(host.emit, 'b', 1)
124+
limit(host.emit, 'b', 2, 2)
125+
limit(host.emit, 'b', 3)
126+
const ended = await host.ended()
127+
assert.deepEqual(ended.map(entry => [entry.limited, entry.silent ?? null, entry.streak ?? null]), [[true, true, 1], [true, null, null], [true, true, 1]])
128+
assert.equal(failureLines({ ended }, 'b', identity).length, 2, 'only the turns that wrote nothing draw a line')
129+
})
130+
131+
test('#2679: a failure and a cut-off turn share the streak, and a finished turn resets it', async (t) => {
132+
const host = await start()
133+
t.after(host.stop)
134+
die(host.emit, 'b', 1)
135+
limit(host.emit, 'b', 2)
136+
host.emit('b', 'turn/start', { turn: 3 })
137+
host.emit('b', 'assistant/message', { turn: 3, message: { content: [{ type: 'text', text: 'ok' }] } })
138+
host.emit('b', 'turn/end', { turn: 3, reason: { kind: 'completed' } })
139+
limit(host.emit, 'b', 4)
140+
const ended = await host.ended()
141+
// The same place's silent turns in a row: one provider refusal and one cut-off turn read as two.
142+
assert.deepEqual(ended.map(entry => [entry.limited ?? null, entry.failed === undefined ? null : 'failed', entry.silent ?? null, entry.streak ?? null]),
143+
[[null, 'failed', true, 1], [true, null, true, 2], [null, null, null, null], [true, null, true, 1]])
144+
})
145+
95146
test('#1044: a turn that wrote text then failed adds no line', async (t) => {
96147
const host = await start()
97148
t.after(host.stop)

‎core/test/flybot-feed.test.js‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,25 @@ test('a Bot turn with a user trigger and a tool folds into turn and tool events'
9191
assert.ok(!JSON.stringify(feed).includes('hello'))
9292
})
9393

94+
// #2679: DSH ends a turn at the model's output ceiling with `reason.kind === 'max-tokens'`, which is
95+
// neither a failure nor a normal finish. The three-value outcome stays (the turn did end), and
96+
// `limited` is the fact a consumer acts on, so every reader is no longer told "finished normally".
97+
test('#2679: a turn cut off at the output limit says so instead of a bare completed', async (t) => {
98+
const { feed, emit, stop } = await boot()
99+
t.after(stop)
100+
emit('a', 'turn/start', { turn: 4 }, 2000)
101+
emit('a', 'assistant/message', { turn: 4, message: { content: [{ type: 'text', text: 'half a sen' }] } }, 2100)
102+
emit('a', 'turn/end', { turn: 4, reason: { kind: 'completed' } }, 2200)
103+
emit('a', 'turn/start', { turn: 5 }, 2300)
104+
emit('a', 'turn/end', { turn: 5, reason: { kind: 'max-tokens' } }, 2400)
105+
const [whole, cut] = only(feed, 'turn.end')
106+
assert.equal(whole.outcome, 'completed')
107+
assert.equal(whole.limited, undefined, 'a normal finish stays unmarked')
108+
assert.equal(cut.outcome, 'completed')
109+
assert.equal(cut.limited, true)
110+
assert.equal(cut.errorCode, undefined, 'a cut-off turn carries no error code')
111+
})
112+
94113
test('a Bot-to-Bot send reaches the feed with the exchange id and the reply links back by replyTo', async (t) => {
95114
const { fb, feed, emit, agents, stop } = await boot()
96115
t.after(stop)

‎core/test/rounds.test.js‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1751,6 +1751,37 @@ test('a face room\'s roster and routing rule stand in its section; a routed turn
17511751
assert.ok(turns.some(turn => turn.sessionId === hosted.sessions[writer]))
17521752
})
17531753

1754+
// #2679: a member whose turn DSH cut off at the output ceiling before it wrote a line did not read the
1755+
// line and choose silence — it never answered. The round now records that slot `missed` (the value
1756+
// `runTurn` already uses for a member that ran out of time), while a completed turn that writes nothing
1757+
// is still the pass it chose.
1758+
test('#2679: a member cut off at the output limit is missed, not passed', async (t) => {
1759+
const { call, emit, userMessage, relay, stop, sent } = start()
1760+
t.after(stop)
1761+
const room = await team(call, sent)
1762+
const state = (await call('state', {})).value
1763+
const chiefName = state.bots.find(bot => bot.id === state.mainBotId).name
1764+
await emit(room.id, userMessage('Tell me the plan'))
1765+
const running = relay(room.id, '2679-limit')
1766+
// The admin's turn starts, runs, and is cut off before it writes anything.
1767+
const first = await turnFor(sent)
1768+
const answering = async (turn, sessionId, exchangeId, reason) => {
1769+
await emit(sessionId, { type: 'turn/start', data: { turn } })
1770+
await emit(sessionId, { type: 'user/message', data: { source: { kind: 'bot', role: 'room', exchangeId }, content: [{ type: 'text', text: 'your turn' }] } })
1771+
await emit(sessionId, { type: 'turn/end', data: { reason } })
1772+
}
1773+
await answering(1, first.sessionId, first.message.source.exchangeId, { kind: 'max-tokens' })
1774+
// The next member reads the cut-off one as "(no answer)": the pass it used to look like said the
1775+
// member had read the line and had nothing to add.
1776+
const second = await turnFor(sent, first)
1777+
assert.match(promptOf(second), new RegExp(`${chiefName} \\(no answer\\)`))
1778+
await answering(2, second.sessionId, second.message.source.exchangeId, { kind: 'completed' })
1779+
const done = JSON.parse(await running)
1780+
assert.deepEqual(done.replies.filter(reply => reply.kind === 'missed').map(reply => reply.name), [chiefName],
1781+
'the cut-off member is the only one recorded as missed')
1782+
assert.equal(done.replies.filter(reply => reply.kind === 'missed').length, 1, 'a completed turn with no text is a pass, not a miss')
1783+
})
1784+
17541785
// #913: the closing report wrote a member the line called with `@` and that never answered into the
17551786
// same sentence as a member that read the line and answered [PASS], so the person could not tell "he
17561787
// had nothing to add" from "he was addressed and said nothing". The round already knows which of its

‎core/types/flybot.d.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,12 @@ export interface TurnEnded extends FeedPlace {
112112
errorCode?: string
113113
/** True when the turn never reported its end and a newer turn closed it. */
114114
interrupted?: boolean
115+
/**
116+
* True when DSH ended the turn at the model's output ceiling (`reason.kind` `max-tokens`): the
117+
* turn ended and did not fail, so `outcome` stays 'completed', but the text it wrote is incomplete
118+
* and a turn that wrote none stopped before it answered.
119+
*/
120+
limited?: boolean
115121
/**
116122
* `provider/model` of the turn's latest request header: the model it ran on. Absent for a
117123
* turn that made no model call, for place 'remote', and when DSH logged no header.

0 commit comments

Comments
 (0)