perf(chat): widen the streaming flush window with the reply it re-renders

[Adam-Dalloul]

For #589: at around 300 tokens a second the whole UI stops responding, on a machine with plenty of headroom.

I measured where the per-batch time goes rather than guessing, driving a realistic chunk stream through the real code.

The store and the timeline are not it. setLiveMessage plus computeTimeline costs 0.03 to 0.12 ms a batch over a 2000-batch turn, and running the message adapter on top brings that to 0.15 to 1.0 ms. Rendering the reply is three orders of magnitude above that. Every batch replaces the live message, so the prose run it appended to goes to the markdown renderer again whole: normalized, re-lexed into blocks, re-highlighted, re-rendered. In the test renderer that is 3.1 ms a batch at 4 KB and 27 ms at 64 KB; the block splitter alone (marked) is 0.7 ms at 4 KB and 6.9 ms at 64 KB. Handing it an unchanged string costs 0.075 ms, so all of it is the run having grown.

The window those batches landed in was a flat 16 ms no matter how long the reply had got. So the work a turn costs rose with the square of its own output while the rate it arrived at stayed put. Replaying 300 tok/s and counting the characters re-rendered across the turn:

                       30 s (35 KB)   120 s (141 KB)
flat 16 ms window          32.5M          518.8M
scaled window              11.1M           59.9M

Sixteen times the work for four times the answer, before. Near linear, after. End to end through the real markdown renderer, 30 seconds of output went from 1803 batches and 1.9 to 3.7 times realtime in render work, to 888 batches and 1.4 to 1.6 times. (Those two numbers move with machine load; the character counts above are deterministic, which is what the test asserts on.)

The window now scales with the run being re-rendered: under 8 KB, which is nearly every reply, it is the same 16 ms as today, and past that each further 8 KB buys one more frame, up to 192 ms. It is sized from the run and not from the whole message, so a reply that has already written 9 KB and then ran a tool is back to a single frame for the block it starts next.

Nothing about what gets delivered changes. The queue merges and dispatches exactly as before, every chunk lands once and in order, and every non-streaming event still flushes it immediately, so a tool card, a permission prompt or the end of a turn never waits on this window.

Still open, and deliberately not in here:

• The per-batch cost itself. The run is re-lexed and re-rendered whole each time. Splitting a streaming reply at block boundaries so only the tail is rebuilt would remove that, but not without changing how a list, a table or a fence spanning the split renders, so it needs its own change.
• The "rendering is off" part of the report. I could not reproduce it and found no chunk reordering or loss in the queue, so I have not tried to fix it. It should stay open as its own thing.

Tests: a new streaming-flush-cadence.test.ts for the window and for what a 300 tok/s turn costs under it, plus two in acp-connections-context.test.tsx driving real events through the provider on fake timers, one checking a long run waits the extra frames and the text arrives exactly as sent, one checking a new run goes back to a single frame.

Composes with the open PRs rather than colliding: #705 and #708 both change computeTimeline / conversation-runtime-store.ts, and #703 is backend. This touches only the queue in acp-connections-context.tsx, which sits above all of them, and it makes their work run fewer times per turn rather than changing what it does.

[xintaofei]

codeg work task 194 is done — #716 (3 files, +928/-106).