Repository navigation
fix(terminal): serve the image transfers a reader actually sends - #819
glitch-cmyk658 wants to merge 6 commits into
Conversation
…hed, and repaint once more Three app-side fixes on top of the merged graphics support: - The text-area answer used the geometry captured at spawn, so a reader that divides it by the cell size laid pages out for a pane that no longer existed. Answer from the grid the session is rendering. - A full-page reader keeps several pages decoded around the cursor; 16 images / 128 MiB let the visible page be evicted without a way to come back. - A decode that finishes between frames could leave the texture unpresented until the next input event, which is why the page needed a click. Request one more frame shortly after the decode completes.
A page reader sends `...,C=1,z=-1,q=1` on every page transfer. `z` only carries meaning together with `o=z`, and `-1` is a signed value the other numeric keys never are, so the whole control was rejected with ENOTSUP and the page was never stored — the reader showed an empty page area. The payload is still checked against the declared pixel size when it is decoded, so ignoring the key removes a false rejection without accepting malformed transfers.
A reader that only knows the graphics protocol sends a container, asks for the features a terminal supports first, and deletes what it placed by cell rather than by id. Each of those used to be refused or silently dropped. - Accept `f=100` containers. The terminal keeps the file and reads the size from its `IHDR` header, so a page no longer has to travel as raw pixels; the frontend inflates it on the decode thread. - Answer `a=q,e=<action>` probes with the keys this terminal honours instead of rejecting the marker, so a client can downgrade before it sends. - Decode payloads without canonical padding and ignore whitespace inside them, which scripts and newline-wrapped writers produce. - Support `d=p`, `d=c`, `d=x`, `d=y`, `d=r` and `d=n` deletes, keeping every other placement alive. - Raise the encoded staging budget to 64 MiB: a 4K page is 33 MiB of base64 while it streams, and the previous 24 MiB refused it mid-transfer. Tests: the graphics unit tests gain twelve cases covering the container header, the feature answers, unpadded payloads and each delete selector.
The reader path has been walked end to end several times now, but only through one document: a bug in crop handling, chunk joining or a delete selector shows up as a blank page and gets diagnosed by eye. This turns the protocol surface into seventeen cases that assert on placements and cells read back from the grid, so a regression fails a test instead of a page. Each case drives the real byte stream through the stream processor and then looks at the terminal: source rectangle, cell budget, channel count, crop window, delete selectors, capability replies, payload tolerance and the pixel budget. The container case builds its own PNG with a stored deflate block, so it needs no fixture file and no extra dependency.
Cost of the image pathNumbers from this branch ( Where the time goesA page goes through three stages: parse+decode in the terminal, hand-off, then create the texture and draw in the app. Only the first and third are ours; the middle is a weak reference to the same buffer, so there is no copy and no round trip.
Measured, isolated (release,
|
| case | payload | ingest |
|---|---|---|
| A4 page, RGB 1060×1500 | 4.77 MB → 6.36 MB base64 | 3.96 ms |
| A4 page, RGBA 1060×1500 | 6.36 MB → 8.48 MB base64 | 5.66 ms |
| 4K frame, RGB 3840×2160 | 24.9 MB → 33.2 MB base64 | 20.68 ms |
| A4 container (PNG), staging only | 2.1 MB | 1.56 ms |
App side:
| case | decode | decode + repack to BGRA |
|---|---|---|
| A4 container 1060×1500 | 19.41 ms | 21.98 ms |
| 4K container 3840×2160 | 105.4 ms | 115.04 ms |
| 4K raw → BGRA | — | 13.69 ms |
So the expensive case is a container that has to be inflated and then repacked: ~115 ms for a 4K page. A raw transfer of the same page is ~35 ms total (20.7 ingest + 13.7 repack).
Measured end to end, one real reader
bookokrat paging 计算机组成结构化方法 (A4, 1060×1500, raw RGB, ~6.4 MB of base64 per page), 20 page turns, timestamps taken inside the app:
- enqueue → draw: 2.9 ms, 3.2 ms, 3.5 ms, 3.5 ms, 3.7 ms, 4.7 ms (six consecutive turns; the rest were re-placements of an already decoded page and produce no new work)
- the page is visible in the frame that follows the decode, so the perceived cost is one frame plus the work above.
The wall-clock from the pty write to the first byte of the app's frame was not measured reliably: the terminal tags the transfer with the client's image id while the app tracks its own placement id, so the two logs do not join without extra plumbing. What is known is the ingest number above and that the app never re-decodes the same page twice (re-placements reuse the cached texture).
Memory
| budget | before | after |
|---|---|---|
| app cache | 16 images / 128 MiB | 32 images / 256 MiB |
| terminal store | 16 images / 128 MiB | 16 images / 128 MiB |
| staging for one transfer | 24 MiB | 64 MiB |
| worst case resident | ≈ 280 MiB | ≈ 448 MiB |
An A4 page costs ≈ 11 MB resident (4.77 MB in the terminal, 6.36 MB as a BGRA texture), so the terminal's store is what caps first at ≈ 26 pages / ≈ 290 MB. A 4K transfer peaks at ≈ 58 MB transient (33 MB of base64 plus 25 MB decoded) and the staging cap is what makes it possible at all — before this branch a 4K page failed with E2BIG before it was ever stored.
What I would do next, if the 115 ms container case matters
Inflate straight into the texture format the app draws (BGRA), which removes the repack pass and roughly halves the container cost. That is a separate change and not part of this PR.
the intel runner timed out signing in a powershell pointer fixture, which turned the two reporting jobs red. nothing in this commit changes the tree; it only asks the matrix for one more attempt.
Two commits that together make the APC image path work against a real page reader.
The transfer a reader sends is refused before it is stored
A reader marks "nothing to undo" with
z=-1in the header of the crop it is about to place.zonly describes payload packing wheno=zis present, which this terminal does not implement, so that value must not decide whether a transfer is accepted. Parsing it is also wrong here: every other numeric key is unsigned, while this one carries -1, soz=-1failed as a parse error and the page was dropped before it reached storage. The field is now ignored; the payload length is still checked when the transfer is decoded.The rest of the protocol the reader expects
f=100containers. The terminal keeps the file as sent and reads the geometry from itsIHDRheader; the frontend inflates it on the decode thread, so a page never has to travel as raw pixels. Bounded by a 64 MiB container budget and the existing pixel cap.a=q,e=<action>answers with the keys this terminal honours (e=T,C=1,f=100,...) instead ofENOTSUP. That answer is how a client discovers the container format before sending one, and it can downgrade otherwise.d=p,d=c,d=x,d=y,d=randd=nin addition tod=a/d=i; the uppercase form still frees the pixel data.Tests
cargo test -p nebula_terminal --lib- 384 passed, 2 failed; the two failures are the pre-existingtty::windows::proxy_tests, which need a real consolecargo test -p nebula --features gpui-shell --bin pebrel inline_image::tests- 19 passedcargo test -p nebula_terminal --test grid_images- 12 passedcargo fmt --allandpython scripts/check_architecture.py- both cleana=T,f=100renders inside its 20x10 cell box; a 584 page PDF still renders through the same reader; the three feature-query replies were captured from a raw pty:U=1is omitted because it is not implemented, which is what the protocol asks for: echo the queried keys that are supported and non-default.Not implemented here:
t=f/t=t/t=stransport,U=placeholders,o=z, placeholder reuse throughI=and animation frames.