Skip to content

fix(streaming): settle uncancelled streams without terminal events - #986

Open
rergards wants to merge 4 commits into
anima-research:mainfrom
rergards:fix/af907-silent-stream-end
Open

rergards wants to merge 4 commits into
anima-research:mainfrom
rergards:fix/af907-silent-stream-end

Conversation

@rergards

@rergards rergards commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem

An uncancelled stream can exhaust without a terminal event. The agent then stays streaming, reports a completed lifecycle, and ignores a later nudge. Closes #907.

Changes

Settle that end as a failed inference through the existing post-loop owner. Release the current turn, record the failure, and allow the next explicit nudge to start work. Keep cancellation, stream-generation guards and protected usage accounting. Retain the established failure through synchronous trace listeners, including listener-triggered disposal.

Merged current main 36b6dfe9 without rewriting the PR history. The one conflict was final failure logging: both raw error and main's operator-facing errorWords now use the same retained cause. The disposal regression checks both fields.

Tests

Current head d704ac08:

  • TypeScript compiler with emit: PASS.
  • Seven focused compiled lifecycle, usage-accounting, cancellation, stop, quiesce, restart and exhausted-turn-rearm suites: 226 pass / 0 fail / 0 skipped.
  • Negative control: an intentionally wrong operator-facing failure reason makes the disposal regression fail; the restored candidate passes.
  • Independent Astra code review of the exact current-main diff: PASS.
  • All 16 current-head GitHub CI checks: PASS, including the three dependency-floor suites. Main Linux suite: 10814 pass / 0 fail / 63 skipped / 1 todo. Separate unforced shutdown suite: 133 pass / 0 fail / 0 skipped.

Original and previous follow-up evidence remains valid for unchanged behavior: the liveness regression failed on unfixed base 0c570d8; two review regressions failed on 64062c09. The original cookbook cycle used Astra plan, Opus plan review/writer and independent Astra code review. This current bounded merge correction was coordinated directly; it has no new cookbook-cycle verdict.

Review follow-up

  • 2, 6: capture the silent-end cause before trace callbacks; retain it in protected final logging and use the same diagnostic in settlement, trace and log.
  • 3: call the existing nontransient rearm disposition. It explains why rearming ends and schedules no new model call.
  • 4: assert exactly one failure trace, including logging-failure and listener-disposal cases.
  • 1, 5: confirmed in the unchanged base. The ownership-lost iterator-close race and iterator-throw logging-order issue remain pre-existing residuals outside this patch.

Related work and limits

AI-assisted using Claude Code and OpenAI Codex.

rergards and others added 2 commits November 7, 2026 00:43
Settle an unexpected iterator end as a failed inference, release the existing
turn, and allow the next explicit nudge. Preserve cancellation and stream
generation guards. Add regression coverage and a release fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Log the silent failure through the existing finally owner so usage, the
established diagnostic, and logging-error isolation stay intact. Cover a
zero-event logging write failure without retrying or leaking its turn.

Co-Authored-By: GPT-6 (OpenAI Codex) <noreply@openai.com>
@rergards
rergards marked this pull request as ready for review November 7, 2026 00:15
@anima-review

anima-review Bot commented Nov 7, 2026

Copy link
Copy Markdown

Automatic review of 64062c09b7fa.

About these findings

The findings below came from reading this PR and relevant code; not a holistic understanding of the project's functional intent and its architecture choices. Finding precision is higher for flagging issues than identifying ideal changes/fixes. This review prioritizes recall - minor findings (in particular) may be nits.

Before actioning a finding, consider the relevant code from first principles given the functional+architectural intent. In particular, when considering adding machinery/logic, scrutinize whether the incremental value of such pays rent for the complexity it adds to the codebase - kudzu has a direct negative impact on our future ability to work with the codebase.

Consider explicitly mentioning which findings you accept/decline (by code-formatted numerical id) and why; this enables future rounds of automated review to take your notes into account.

Findings, most severe first:

  • 1. minor at src/framework.ts:27369, src/framework.ts:27566, src/framework.ts:27621, src/framework.ts:25656, src/framework.ts:27582, src/framework.ts:27524: A stream that loses ownership while awaiting next() and then closes skips ownership-loss handling at loop exit.
    • Failing case: If a cancelled or replaced stream's pending next() returns done after a successor has taken over, the loop body never sets generationLost and the post-loop guard skips settlement; finally therefore emits the default completed lifecycle and calls stopTyping even though the old frame no longer owns the agent.
  • 2. minor at src/framework.ts:27398, src/framework.ts:27524, src/framework.ts:27529, src/framework.ts:11705: The deferred log write attributes an already-settled silent failure to a later disposal or supersession.
    • Failing case: An inference:exhausted trace listener synchronously calls stopAgent for a non-default agent; the silent-end branch has already settled the failure, but finally sees the agent removed and records 'Stream abandoned: its agent was disposed or replaced' instead of the actual missing-terminal-event cause.
  • 3. minor at src/framework.ts:27398, src/framework.ts:6718, src/framework.ts:6722: The silent-end path omits the exhausted-turn rearm disposition that sibling failure paths emit.
    • Failing case: With turn rearming enabled, let a transient failure rearm a turn and let the rearmed stream end silently; unlike a non-user abort, this branch never calls rearmExhaustedTurn with a non-transient failure, so the operator receives no 'turn not re-armed' explanation for why the retry sequence stopped.
  • 4. minor at test/inference-lifecycle-failed.test.ts:300, test/inference-lifecycle-failed.test.ts:334: The silent-failure tests never assert that exactly one failure trace is emitted.
    • Failing case: An implementation emitting both inference:failed and inference:exhausted, or emitting inference:exhausted twice, still satisfies the first test's trace assertions and is not detected by the log-write-failure test.
  • 5. minor at src/framework.ts:27455, src/framework.ts:27501, src/framework.ts:27508, src/framework.ts:27582: The iterator-throw catch sets the failed phase after an unprotected log write, so a write failure reports completed.
    • Failing case: A stream iterator throws and the inference-log store append also throws, for example because storage is unavailable: finally still emits the lifecycle terminal, but reports completed despite the failure trace and no successful inference.
  • 6. minor at src/framework.ts:27389, src/framework.ts:27401, src/framework.ts:27524, src/framework.ts:27533, src/framework.ts:14509: The trace and error say 'terminal event' but the inference log says 'final response', which breaks correlation by reason.
    • Failing case: After an uncancelled stream closes without a terminal event, the trace, failure-streak record and ops-facing failure reason say Stream ended without a terminal event, but queryInferenceLogs({ pattern: 'Stream ended without a terminal event' }) does not return that failure because its row says Stream ended without a final response.

@nissa-seru

Copy link
Copy Markdown
Contributor

/context-deltas

@anima-review

anima-review Bot commented Nov 7, 2026

Copy link
Copy Markdown

Automatic context view of 64062c09b7fa merged onto 0c570d8f2d8c.

What a context view runs and what its checks judge: CHECKS.md.

The records of the 27 runs don't differ between the base and this head, so no request changed.

Held the same in base and head: connectome-host 7278e9c58fd7, chronicle 0.5.1, context-manager 0.24.0, membrane 0.8.0.

Keep the established cause in protected final logging even when a trace
listener disposes the agent. Explain why a rearmed silent failure stops,
and assert single failure notifications without adding another retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to anima-research/context-view that referenced this pull request Nov 9, 2026
…ay out

Four kinds of change a view couldn't show, since no run reached their paths:

- Hcont/W13 sends to its turn's own channel with channel_publish and then
  speaks (step `sent`), so the send holds its words as drafts and the
  continuation's `[continue]` row takes its held form. Its fate is a receipt
  naming no channel with drafts held, and one continuation. The step says its
  speech is held for an explicit send, so check 1 leaves the hold alone.
- W12 under H5 publishes words the fixture server refuses to post, then finds
  them with the history module's search and reads them back in text form
  (`hist.fail`, `hist.read2`), where the extract marks the send `, error`.
  They're judged as the first send's are: the search's result and the
  extract's text each carry the words. The mark isn't checked, since the
  record shows it. Under H5c, whose budget the host fits to the model's
  window, the added steps make the store compress, so they run under H5
  alone; nothing in them depends on a setup's values.
- W12 under H5, where a failed judge wakes the agent, adds a judged batch
  (`judge.c2`) of one message addressed to the agent, which the agent reads
  and answers in w12-judge (`judge.read2`) before its judge fails
  (`judge.err2`). It's judged as an answered batch is: the judge's one request
  fails, and the batch starts one turn at most, its reply a spare, since
  anima-research/agent-framework#972 drops that wake. judge.c1's unread
  failed batch stays as it was, so a failed judge's wake stays reached once
  that pull request merges. The reading turn's publish carries no canary: check 13 pairs a
  compile with its request by the newest canary each carries, and a record
  names none inside a call's input, so the judge's wake, which adds none,
  would read as unpaired.
- Sources of private input that anima-research/agent-framework#190 names,
  each after a message typed at the console: W12 under H5 spawns a subagent
  without waiting, so its result wakes a turn (`priv.spawn`); W16's later
  turns, the unstick's among them, follow a console message after its first
  turn (`pre.con`); and W17 adds a sleep's end and an operator's nudge. Where
  each turn's speech goes shows in the record, and no check judges it.

CHECKS.md describes each, and adds what each workload leaves out and why:
under W12, H5b's read failed batch, where nothing wakes either way, and a
fork's result after console input; under W13,
anima-research/agent-framework#940's onSpeechRouted receipt and its ended
speaker's notice, and anima-research/agent-framework#947's coalesced
acceptance; under W17, a tune-out's cancel after console input, and
anima-research/agent-framework#190's first start after a restart and its
inference.request turn; under W18, anima-research/context-manager#369's
changing viewFilter and its prompt-cache claim; and, in a new list of paths no
run reaches, anima-research/agent-framework#986's stream with no terminal
event. The fork's result and the tune-out's cancel each derive a fork inside
the agent's own call, a shape check 11 doesn't yet read, so a run reaching
either flags check 11 at agent-framework main aa134e75.
nissa-seru pushed a commit to anima-research/context-view that referenced this pull request Nov 9, 2026
…nother

Each view reads one profiled drive of each run. On
anima-research/agent-framework#986, six views agreed on every changed line.
Some host lines still ran in some views and not others: src/framework.ts
5995-5996, a 1 s sync timer's body, and 1882-1890, isTurnContinuation and
passesQuiesce, reached on quiesce paths and batch splits, by the status
read's count of gated requests, and by a transient failure's re-arm. The
records stayed byte-identical, so identical records don't mean identical
execution.

Co-authored-by: Desmond-2579 <desmond-2579@commons.astra.invalid>
nissa-seru pushed a commit to anima-research/context-view that referenced this pull request Nov 9, 2026
…ay out

Four kinds of change a view couldn't show, since no run reached their paths:

- Hcont/W13 sends to its turn's own channel with channel_publish and then
  speaks (step `sent`), so the send holds its words as drafts and the
  continuation's `[continue]` row takes its held form. Its fate is a receipt
  naming no channel with drafts held, and one continuation. The step says its
  speech is held for an explicit send, so check 1 leaves the hold alone.
- W12 under H5 publishes words the fixture server refuses to post, then finds
  them with the history module's search and reads them back in text form
  (`hist.fail`, `hist.read2`), where the extract marks the send `, error`.
  They're judged as the first send's are: the search's result and the
  extract's text each carry the words. The mark isn't checked, since the
  record shows it. Under H5c, whose budget the host fits to the model's
  window, the added steps make the store compress, so they run under H5
  alone; nothing in them depends on a setup's values.
- W12 under H5, where a failed judge wakes the agent, adds a judged batch
  (`judge.c2`) of one message addressed to the agent, which the agent reads
  and answers in w12-judge (`judge.read2`) before its judge fails
  (`judge.err2`). It's judged as an answered batch is: the judge's one request
  fails, and the batch starts one turn at most, its reply a spare, since
  anima-research/agent-framework#972 drops that wake. judge.c1's unread
  failed batch stays as it was, so a failed judge's wake stays reached once
  that pull request merges. The reading turn's publish carries no canary:
  check 13 validates each compile it pairs with a request by the newest
  canary each carries, and a record names none inside a call's input, so
  with one there the judge's wake, which adds none, would read as a pair
  that disagrees, and check 13 would judge nothing.
- Sources of private input that anima-research/agent-framework#190 names,
  each after a message typed at the console: W12 under H5 spawns a subagent
  without waiting, so its result wakes a turn (`priv.spawn`); W16's later
  turns, the unstick's among them, follow a console message after its first
  turn (`pre.con`); and W17 adds a sleep's end and an operator's nudge. Where
  each turn's speech goes shows in the record, and no check judges it.

CHECKS.md describes each, and adds what each workload leaves out and why:
under W12, H5b's read failed batch, where nothing wakes either way, and a
fork's result after console input; under W13,
anima-research/agent-framework#940's onSpeechRouted receipt and its ended
speaker's notice, and anima-research/agent-framework#947's coalesced
acceptance; under W17, a tune-out's cancel after console input, and
anima-research/agent-framework#190's first start after a restart and its
inference.request turn; under W18, anima-research/context-manager#369's
changing viewFilter and its prompt-cache claim; and, in a new list of paths no
run reaches, anima-research/agent-framework#986's stream with no terminal
event. The fork's result and the tune-out's cancel each derive a fork inside
the agent's own call, a shape check 11 doesn't yet read, so a run reaching
either flags check 11 at agent-framework main aa134e75.
nissa-seru pushed a commit to anima-research/context-view that referenced this pull request Nov 9, 2026
…nother

Each view reads one profiled drive of each run. On
anima-research/agent-framework#986, six views agreed on every changed line.
Some host lines still ran in some views and not others: src/framework.ts
5995-5996, a 1 s sync timer's body, and 1882-1890, isTurnContinuation and
passesQuiesce, reached on quiesce paths and batch splits, by the status
read's count of gated requests, and by a transient failure's re-arm. The
records stayed byte-identical, so identical records don't mean identical
execution.

Co-authored-by: Desmond-2579 <desmond-2579@commons.astra.invalid>
@rufalupa666-netizen

Copy link
Copy Markdown

Read the framework.ts diff and the four tests. Two notes from the #980 side rather than findings against the change:

  1. Boundary with On the Responses path (openai-codex), JSON ping events can keep a contentless call open indefinitely; #378's bounds don't cover it #980. This settles a stream that ends with no terminal event. A stream the Responses path keeps open with JSON ping events never reaches this branch — the iterator doesn't close — so On the Responses path (openai-codex), JSON ping events can keep a contentless call open indefinitely; #378's bounds don't cover it #980 stays open after this merges; worth a word in the changelog so nobody reads "silent stream end" as covering it. For what it's worth, on our live openai-codex host (agent-framework 0.32.0, ~45 streaming calls since Oct 28) we have seen neither a silent end nor a ping-held stream; both remain unreproduced here.

  2. Interplay with fix(framework): an exhausted turn that did nothing writes no [inference-failed] note #985 (an exhausted turn that did nothing writes no [inference-failed] note), open on the same region. A silent end is exactly an exhausted turn that produced nothing, so if fix(framework): an exhausted turn that did nothing writes no [inference-failed] note #985 lands first the operator-visible line for this failure is whatever rearmExhaustedTurn(…, false) says to a re-armed turn's operator plus the llm-calls entry — which may be the intent, but the two PRs should agree on where a silent end shows up on stderr. Flagging so whichever merges second accounts for the other.

The silentFailure retention through the trace callback (test 4) is a good catch; the settle-as-aborted ordering reads consistent with the aborted case.

Retain the established silent failure in both raw and operator-facing log fields. Clarify that ping-held streams remain outside issue907.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
@rergards

Copy link
Copy Markdown
Contributor Author

Updated in d704ac08, merged with current main 36b6dfe9 without rewriting the existing PR history.

  1. Accepted the boundary clarification. The changelog now explicitly says this fix requires the iterator to close. A contentless stream kept open by JSON ping events remains outside it, so issue On the Responses path (openai-codex), JSON ping events can keep a contentless call open indefinitely; #378's bounds don't cover it #980 stays open. No watchdog or new timeout option was added here.

  2. The silent-end path emits inference:exhausted and writes its failure through the protected inference-log owner. For an already rearmed turn, the existing nontransient rearm disposition emits the explanation that the turn will not be rearmed. This path does not call the common context-marker-writing failure path and does not promise a general [inference-failed] stderr line for every silent end. PR fix(framework): an exhausted turn that did nothing writes no [inference-failed] note #985 remains the owner of that common marker policy. Whichever merges second should retain this boundary; this branch neither copies nor depends on fix(framework): an exhausted turn that did nothing writes no [inference-failed] note #985.

The current-main merge exposed one logging conflict: main added operator-facing errorWords beside raw error. Both now retain the established silent failure even if the trace listener subsequently disposes the agent. The existing listener-disposal regression asserts both fields. Its wrong-operator-reason negative control fails, and the restored candidate passes.

Validation: compiler PASS; seven focused compiled suites, 226 pass / 0 fail / 0 skipped; independent Astra exact-diff review PASS. All 16 current-head CI checks passed, including three dependency-floor suites. Main Linux suite: 10814 pass / 0 fail / 63 skipped / 1 todo; separate unforced shutdown suite: 133 pass / 0 fail / 0 skipped.

No live-provider silent end or ping-held stream was exercised. The tests use the reported iterator contract and synthetic transports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A stream that ends with no events, uncancelled, leaves its agent streaming, and a second nudge 3 s later starts no model call

3 participants