Skip to content

fix(terminal): serve the image transfers a reader actually sends - #819

Open
glitch-cmyk658 wants to merge 6 commits into
Kuddev:mainfrom
glitch-cmyk658:fix/main-frame-presentation
Open

glitch-cmyk658 wants to merge 6 commits into
Kuddev:mainfrom
glitch-cmyk658:fix/main-frame-presentation

Conversation

@glitch-cmyk658

@glitch-cmyk658 glitch-cmyk658 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Two commits that together make the APC image path work against a real page reader.

The transfer a reader sends is refused before it is stored

A reader marks "nothing to undo" with z=-1 in the header of the crop it is about to place. z only describes payload packing when o=z is present, which this terminal does not implement, so that value must not decide whether a transfer is accepted. Parsing it is also wrong here: every other numeric key is unsigned, while this one carries -1, so z=-1 failed as a parse error and the page was dropped before it reached storage. The field is now ignored; the payload length is still checked when the transfer is decoded.

The rest of the protocol the reader expects

  • f=100 containers. The terminal keeps the file as sent and reads the geometry from its IHDR header; the frontend inflates it on the decode thread, so a page never has to travel as raw pixels. Bounded by a 64 MiB container budget and the existing pixel cap.
  • Feature queries. a=q,e=<action> answers with the keys this terminal honours (e=T,C=1,f=100,...) instead of ENOTSUP. That answer is how a client discovers the container format before sending one, and it can downgrade otherwise.
  • Payload tolerance. Base64 without canonical padding decodes, and whitespace inside a payload is stripped instead of dropping the whole sequence silently.
  • Deletes. d=p, d=c, d=x, d=y, d=r and d=n in addition to d=a/d=i; the uppercase form still frees the pixel data.
  • Encoded staging budget. 24 MiB to 64 MiB: a 4K page is 33 MiB of base64 while it streams, and the old cap refused it mid-transfer.

Tests

  • cargo test -p nebula_terminal --lib - 384 passed, 2 failed; the two failures are the pre-existing tty::windows::proxy_tests, which need a real console
  • the graphics tests inside that run - 38 matched, twelve of them new: the container header, a malformed container, an unpadded payload, the feature answers, and each new delete selector
  • cargo test -p nebula --features gpui-shell --bin pebrel inline_image::tests - 19 passed
  • cargo test -p nebula_terminal --test grid_images - 12 passed
  • cargo fmt --all and python scripts/check_architecture.py - both clean
  • live, on the built binary: a real PNG sent as a=T,f=100 renders inside its 20x10 cell box; a 584 page PDF still renders through the same reader; the three feature-query replies were captured from a raw pty:
a=q,i=31,e=T,U=1,C=1         -> i=31,e=T,C=1;OK
a=q,i=32,e=f                 -> i=32;ENOTSUP
a=q,i=33,e=T,f=100,c=20,r=10 -> i=33,e=T,c=20,r=10,f=100;OK

U=1 is omitted because it is not implemented, which is what the protocol asks for: echo the queried keys that are supported and non-default.

Not implemented here: t=f/t=t/t=s transport, U= placeholders, o=z, placeholder reuse through I= and animation frames.

…hed, and repaint once more

Three app-side fixes on top of the merged graphics support:

- The text-area answer used the geometry captured at spawn, so a reader that
  divides it by the cell size laid pages out for a pane that no longer existed.
  Answer from the grid the session is rendering.
- A full-page reader keeps several pages decoded around the cursor; 16 images /
  128 MiB let the visible page be evicted without a way to come back.
- A decode that finishes between frames could leave the texture unpresented
  until the next input event, which is why the page needed a click. Request one
  more frame shortly after the decode completes.
A page reader sends `...,C=1,z=-1,q=1` on every page transfer. `z` only
carries meaning together with `o=z`, and `-1` is a signed value the other
numeric keys never are, so the whole control was rejected with ENOTSUP and
the page was never stored — the reader showed an empty page area.

The payload is still checked against the declared pixel size when it is
decoded, so ignoring the key removes a false rejection without accepting
malformed transfers.
A reader that only knows the graphics protocol sends a container, asks for
the features a terminal supports first, and deletes what it placed by cell
rather than by id. Each of those used to be refused or silently dropped.

- Accept `f=100` containers. The terminal keeps the file and reads the size
  from its `IHDR` header, so a page no longer has to travel as raw pixels;
  the frontend inflates it on the decode thread.
- Answer `a=q,e=<action>` probes with the keys this terminal honours instead
  of rejecting the marker, so a client can downgrade before it sends.
- Decode payloads without canonical padding and ignore whitespace inside
  them, which scripts and newline-wrapped writers produce.
- Support `d=p`, `d=c`, `d=x`, `d=y`, `d=r` and `d=n` deletes, keeping every
  other placement alive.
- Raise the encoded staging budget to 64 MiB: a 4K page is 33 MiB of base64
  while it streams, and the previous 24 MiB refused it mid-transfer.

Tests: the graphics unit tests gain twelve cases covering the container
header, the feature answers, unpadded payloads and each delete selector.
@glitch-cmyk658 glitch-cmyk658 changed the title fix(terminal): store image transfers a reader marks with z=-1 fix(terminal): serve the image transfers a reader actually sends Oct 30, 2026
The reader path has been walked end to end several times now, but only
through one document: a bug in crop handling, chunk joining or a delete
selector shows up as a blank page and gets diagnosed by eye. This turns the
protocol surface into seventeen cases that assert on placements and cells
read back from the grid, so a regression fails a test instead of a page.

Each case drives the real byte stream through the stream processor and then
looks at the terminal: source rectangle, cell budget, channel count, crop
window, delete selectors, capability replies, payload tolerance and the
pixel budget. The container case builds its own PNG with a stored deflate
block, so it needs no fixture file and no extra dependency.
@glitch-cmyk658

Copy link
Copy Markdown
Contributor Author

Cost of the image path

Numbers from this branch (8ff0d574) on Windows 11, release build, one machine — they are for order of magnitude, not a benchmark suite.

Where the time goes

A page goes through three stages: parse+decode in the terminal, hand-off, then create the texture and draw in the app. Only the first and third are ours; the middle is a weak reference to the same buffer, so there is no copy and no round trip.

stage what it covers
ingest base64 decode, container inflate, size checks, placement
hand-off Weak<Vec<u8>> to the app — no copy
upload decoded pixels → texture, one row loop per frame

Measured, isolated (release, --release, median of 10)

Terminal side, raw payload in one call:

case payload ingest
A4 page, RGB 1060×1500 4.77 MB → 6.36 MB base64 3.96 ms
A4 page, RGBA 1060×1500 6.36 MB → 8.48 MB base64 5.66 ms
4K frame, RGB 3840×2160 24.9 MB → 33.2 MB base64 20.68 ms
A4 container (PNG), staging only 2.1 MB 1.56 ms

App side:

case decode decode + repack to BGRA
A4 container 1060×1500 19.41 ms 21.98 ms
4K container 3840×2160 105.4 ms 115.04 ms
4K raw → BGRA — 13.69 ms

So the expensive case is a container that has to be inflated and then repacked: ~115 ms for a 4K page. A raw transfer of the same page is ~35 ms total (20.7 ingest + 13.7 repack).

Measured end to end, one real reader

bookokrat paging 计算机组成结构化方法 (A4, 1060×1500, raw RGB, ~6.4 MB of base64 per page), 20 page turns, timestamps taken inside the app:

  • enqueue → draw: 2.9 ms, 3.2 ms, 3.5 ms, 3.5 ms, 3.7 ms, 4.7 ms (six consecutive turns; the rest were re-placements of an already decoded page and produce no new work)
  • the page is visible in the frame that follows the decode, so the perceived cost is one frame plus the work above.

The wall-clock from the pty write to the first byte of the app's frame was not measured reliably: the terminal tags the transfer with the client's image id while the app tracks its own placement id, so the two logs do not join without extra plumbing. What is known is the ingest number above and that the app never re-decodes the same page twice (re-placements reuse the cached texture).

Memory

budget before after
app cache 16 images / 128 MiB 32 images / 256 MiB
terminal store 16 images / 128 MiB 16 images / 128 MiB
staging for one transfer 24 MiB 64 MiB
worst case resident ≈ 280 MiB ≈ 448 MiB

An A4 page costs ≈ 11 MB resident (4.77 MB in the terminal, 6.36 MB as a BGRA texture), so the terminal's store is what caps first at ≈ 26 pages / ≈ 290 MB. A 4K transfer peaks at ≈ 58 MB transient (33 MB of base64 plus 25 MB decoded) and the staging cap is what makes it possible at all — before this branch a 4K page failed with E2BIG before it was ever stored.

What I would do next, if the 115 ms container case matters

Inflate straight into the texture format the app draws (BGRA), which removes the repack pass and roughly halves the container cost. That is a separate change and not part of this PR.

the intel runner timed out signing in a powershell pointer fixture,
which turned the two reporting jobs red. nothing in this commit changes
the tree; it only asks the matrix for one more attempt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant