Repository navigation
Add desktop environment SDK client - #423
Conversation
Sandboxes need a public API for desktop environments so Workers can manage desktops, capture screenshots, and stream VNC. The Dockerfile gains a desktop build stage with the required system dependencies.
🦋 Changeset detectedLatest commit: e429e7b The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
OpenCode Review
Solid implementation of desktop environment support for AI computer-use workflows. The architecture follows good patterns - modular DesktopClient, RPC proxy with method allowlist, and proper Docker multi-stage builds.
A few areas need attention:
-
Dockerfile Go version - Line 31 uses unreleased
golang:1.24-bookworm. Go 1.24 isn't released yet (current is 1.23). Usegolang:1.23-bookworminstead to avoid build failures. -
Error handling inconsistencies - DesktopClient has mixed patterns where some methods (like
start(),stop()) catch and rethrow errors while others let errors bubble up naturally. Pick one approach for consistency - I'd recommend letting errors bubble up since BaseHttpClient already handles logging. -
Code duplication - Base64 conversion logic is duplicated between
screenshot()andscreenshotRegion(). Consider extracting to a private helper method. -
Missing tests - No unit tests for the substantial DesktopClient. Consider adding tests for the core methods.
-
Unrelated changes - The backup curl timeout reductions (lines 3243, 3281) appear unrelated to desktop functionality. Consider moving to a separate PR focused on backup performance.
Overall the desktop functionality is well-designed and the changeset properly focuses on user benefits. The RPC proxy approach is particularly elegant for avoiding exposure of internal container methods.
commit: |
🐳 Docker Images PublishedDefault: FROM cloudflare/sandbox:0.0.0-pr-423-b317445With Python: FROM cloudflare/sandbox:0.0.0-pr-423-b317445-pythonWith OpenCode: FROM cloudflare/sandbox:0.0.0-pr-423-b317445-opencodeVersion: Use the 📦 Standalone BinaryFor arbitrary Dockerfiles: COPY --from=cloudflare/sandbox:0.0.0-pr-423-b317445 /container-server/sandbox /sandbox
ENTRYPOINT ["/sandbox"]Download via GitHub CLI: gh run download 22617306658 -n sandbox-binaryExtract from Docker: docker run --rm cloudflare/sandbox:0.0.0-pr-423-b317445 cat /container-server/sandbox > sandbox && chmod +x sandbox |
* Add desktop environment tests Unit tests for the container handler, service, and SDK client, plus an E2E test that exercises the full desktop lifecycle through a real deployed worker. * Add desktop viewer example (#425) React + Vite + Tailwind v4 app that demonstrates the desktop environment API with noVNC streaming and viewport-aware resolution.
* Add desktop environment container runtime Enables running a full Linux desktop inside sandbox containers with programmatic screenshot and input control via native FFI. * Add desktop environment SDK client (#423) * Add desktop environment SDK client Sandboxes need a public API for desktop environments so Workers can manage desktops, capture screenshots, and stream VNC. The Dockerfile gains a desktop build stage with the required system dependencies. * Add desktop environment tests (#424) * Add desktop environment tests Unit tests for the container handler, service, and SDK client, plus an E2E test that exercises the full desktop lifecycle through a real deployed worker. * Add desktop viewer example (#425) React + Vite + Tailwind v4 app that demonstrates the desktop environment API with noVNC streaming and viewport-aware resolution. * Fix lint errors in desktop example and biome config * Add desktop E2E test Dockerfile and config generation * Add desktop image to CI build, push, and cleanup workflows * Fix Go build: pin golang.org/x/net to Go 1.24-compatible version go mod tidy resolved golang.org/x/net@v0.51.0 which requires go >= 1.25, breaking the go-builder stage using golang:1.24-bookworm. Pin to v0.50.0 (last Go 1.24-compatible release) and update go directive to match builder. * Fix FFI type mismatch and clickCount handling The Click FFI binding declared 'bool' (1-byte C _Bool) but the Go function expects C.int (4 bytes), causing undefined ABI behavior. Changed to 'int' and pass clickCount through directly so tripleClick emits three rapid single clicks instead of silently degrading to doubleClick. * Guard desktop stop in destroy() on container state desktop.stop() goes through containerFetch which auto-starts sleeping containers. Check ctx.container.running first so destroy() does not wake a container just to immediately tear it down. * Revert accidental backup and token doc changes The desktop branch commit inadvertently changed backup curl from streaming -T to --data-binary (loads full archive into memory), reduced timeouts from 1800s to 300s, removed the local-dev mismatch diagnostic, and changed token docs to show hyphens which the validation regex rejects. Restore all to match main. * Add error resilience to desktop worker and manager Catch stop() failures during start() error recovery so the original error propagates. Add onerror handler to the worker thread so pending promises reject instead of hanging if the worker crashes. * Fix FFI out-pointer semantics and skip stream-url in CI koffi requires koffi.out() annotation on pointer parameters to copy values back from C to JS after the call. Without it, GetScreenSize and GetMousePos always returned zeros because koffi treated int* as input-only. The stream-url E2E test requires preview URL infrastructure (custom domain with wildcard DNS) that CI workers.dev doesn't provide, so skip it with the same pattern used by other port-exposure tests. * Reset manager state on start failure DesktopManager.start() sets state to 'starting' but the catch block relied solely on stop() to reset it to 'inactive'. When stop() itself fails, state remains 'starting' permanently, blocking all subsequent start attempts. Explicitly set state to 'inactive' after cleanup. * Use pure-Go xgb path for GetScreenSize robotgo.GetScreenSize() delegates to C-based XGetMainDisplay() which holds an unsynchronized static Display pointer. In Go's c-shared build mode CGo dispatches from varying OS threads, causing the singleton to silently return zero dimensions. Switch to robotgo.GetDisplayBounds(0) which uses the github.com/kbinani/screenshot pure-Go xgb implementation, matching the existing workaround for the SaveCapture segfault. * Upgrade robotgo to v1.0.1 with uniform error handling Use dedicated v1.0.1 APIs (MouseDown/Up, KeyDown/Up, Type, MultiClick) instead of Toggle/KeyToggle/TypeStr. All Go FFI exports now return error strings via *C.char, and the koffi bindings use HeapStr with a checkError() helper for uniform error propagation. Rename TypeStr→TypeText and SaveCapture→Screenshot to match v1.0.1 naming. Click now takes a count parameter — single, double, and multi-click are handled in Go. The worker-side triple-click loop is removed since Go handles it natively via robotgo.MultiClick.
2 participants
Summary
Public SDK surface for desktop environments. DesktopClient provides typed methods for lifecycle management, screenshot capture, and VNC stream URLs. The Sandbox class gains an RPC proxy that dispatches desktop calls to the container runtime from the previous PR.
Design decisions
Explicit
DESKTOP_METHODSallowlist on the RPC proxy: Desktop methods are routed through acallDesktop()dispatcher rather than exposing them as open RPC stubs viaget(). This prevents accidental exposure of internal container methods and makes the public API surface auditable in one place.Single
callDesktop()dispatch rather than individual Sandbox methods: Adding 22 methods directly to the Sandbox class would bloat it. Instead, the Sandbox exposes desktop functionality through a property that returns the DesktopClient interface, keeping the core class focused on container lifecycle.Dockerfile desktop stage: The desktop build stage extends the base image with Go compilation (for the robotgo wrapper), XFCE4, Xvfb, x11vnc, websockify, and font packages. It's a separate Docker stage so non-desktop sandboxes aren't affected.
This is part 2 of 4 in the desktop environment stack:
main