Repository navigation
Add desktop environment container runtime - #422
Conversation
Enables running a full Linux desktop inside sandbox containers with programmatic screenshot and input control via native FFI.
🦋 Changeset detectedLatest commit: 4899918 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
commit: |
🐳 Docker Images PublishedDefault: FROM cloudflare/sandbox:0.0.0-pr-422-d9fbe74With Python: FROM cloudflare/sandbox:0.0.0-pr-422-d9fbe74-pythonWith OpenCode: FROM cloudflare/sandbox:0.0.0-pr-422-d9fbe74-opencodeVersion: Use the 📦 Standalone BinaryFor arbitrary Dockerfiles: COPY --from=cloudflare/sandbox:0.0.0-pr-422-d9fbe74 /container-server/sandbox /sandbox
ENTRYPOINT ["/sandbox"]Download via GitHub CLI: gh run download 22631865994 -n sandbox-binaryExtract from Docker: docker run --rm cloudflare/sandbox:0.0.0-pr-422-d9fbe74 cat /container-server/sandbox > sandbox && chmod +x sandbox |
| if (!lib && !loadLibrary()) { | ||
| self.postMessage({ id, error: 'Desktop library not available' }); | ||
| return; |
There was a problem hiding this comment.
nit: could throw here to be more consistent with the rest of the method body?
| case '/api/desktop/start': | ||
| return this.handleStart(request, context); | ||
| case '/api/desktop/stop': | ||
| return this.handleStop(request, context); |
There was a problem hiding this comment.
just an idea for future, would we want a /desktop/restart handler too? Below we advise people to restart with sandbox.desktop.stop() and sandbox.desktop.start(), which we could handle in a sandbox.desktop.restart() call along with some failsafes, and more error handling for specifically restarting a desktop?
* Add desktop environment SDK client Sandboxes need a public API for desktop environments so Workers can manage desktops, capture screenshots, and stream VNC. The Dockerfile gains a desktop build stage with the required system dependencies. * Add desktop environment tests (#424) * Add desktop environment tests Unit tests for the container handler, service, and SDK client, plus an E2E test that exercises the full desktop lifecycle through a real deployed worker. * Add desktop viewer example (#425) React + Vite + Tailwind v4 app that demonstrates the desktop environment API with noVNC streaming and viewport-aware resolution.
# Conflicts: # packages/sandbox-container/src/core/container.ts # packages/sandbox-container/src/routes/setup.ts # packages/sandbox/src/clients/sandbox-client.ts # packages/sandbox/src/sandbox.ts # packages/shared/src/errors/codes.ts
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
go mod tidy resolved golang.org/x/net@v0.51.0 which requires go >= 1.25, breaking the go-builder stage using golang:1.24-bookworm. Pin to v0.50.0 (last Go 1.24-compatible release) and update go directive to match builder.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The Click FFI binding declared 'bool' (1-byte C _Bool) but the Go function expects C.int (4 bytes), causing undefined ABI behavior. Changed to 'int' and pass clickCount through directly so tripleClick emits three rapid single clicks instead of silently degrading to doubleClick.
desktop.stop() goes through containerFetch which auto-starts sleeping containers. Check ctx.container.running first so destroy() does not wake a container just to immediately tear it down.
The desktop branch commit inadvertently changed backup curl from streaming -T to --data-binary (loads full archive into memory), reduced timeouts from 1800s to 300s, removed the local-dev mismatch diagnostic, and changed token docs to show hyphens which the validation regex rejects. Restore all to match main.
Catch stop() failures during start() error recovery so the original error propagates. Add onerror handler to the worker thread so pending promises reject instead of hanging if the worker crashes.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
koffi requires koffi.out() annotation on pointer parameters to copy values back from C to JS after the call. Without it, GetScreenSize and GetMousePos always returned zeros because koffi treated int* as input-only. The stream-url E2E test requires preview URL infrastructure (custom domain with wildcard DNS) that CI workers.dev doesn't provide, so skip it with the same pattern used by other port-exposure tests.
DesktopManager.start() sets state to 'starting' but the catch block relied solely on stop() to reset it to 'inactive'. When stop() itself fails, state remains 'starting' permanently, blocking all subsequent start attempts. Explicitly set state to 'inactive' after cleanup.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
robotgo.GetScreenSize() delegates to C-based XGetMainDisplay() which holds an unsynchronized static Display pointer. In Go's c-shared build mode CGo dispatches from varying OS threads, causing the singleton to silently return zero dimensions. Switch to robotgo.GetDisplayBounds(0) which uses the github.com/kbinani/screenshot pure-Go xgb implementation, matching the existing workaround for the SaveCapture segfault.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Use dedicated v1.0.1 APIs (MouseDown/Up, KeyDown/Up, Type, MultiClick) instead of Toggle/KeyToggle/TypeStr. All Go FFI exports now return error strings via *C.char, and the koffi bindings use HeapStr with a checkError() helper for uniform error propagation. Rename TypeStr→TypeText and SaveCapture→Screenshot to match v1.0.1 naming. Click now takes a count parameter — single, double, and multi-click are handled in Go. The worker-side triple-click loop is removed since Go handles it natively via robotgo.MultiClick.
There was a problem hiding this comment.
OpenCode Review
This PR adds desktop environment support to Cloudflare Sandboxes, enabling AI computer-use workflows with a full Linux desktop stack. The implementation is well-architected and follows the established patterns.
Architecture Assessment:
The desktop variant adds a new 4-layer desktop stack (Xvfb → XFCE4 → x11vnc → noVNC) with proper process orchestration through DesktopManager. The Go→FFI→Worker thread architecture for native control is smart - it provides real desktop interaction while keeping the main HTTP server responsive. The separation between DesktopService (business logic), DesktopManager (process lifecycle), and DesktopWorker (native operations) follows good DI patterns.
Implementation Quality:
- The 18-endpoint REST API provides comprehensive desktop control coverage
- Memory management with
HeapStrdisposables prevents C string leaks - Process validation and readiness checking ensures proper startup ordering
- Error handling properly flows from native Go → JavaScript → HTTP responses
- Container builds are properly parallelized in CI
Docker Implementation:
The new desktop target correctly extends runtime-base and includes all necessary X11 dependencies. The Go builder stage produces the required desktop.so shared library. Port 6080 exposure for noVNC streaming is appropriate.
Testing:
Good test coverage for handlers and services with proper mocking. The tests verify request routing, error propagation, and service interaction patterns.
Documentation Impact:
This feature introduces significant new capabilities that will need documentation updates:
- Desktop environment setup and usage guide
- AI computer-use workflow examples
- Screenshot/input automation patterns
- noVNC streaming setup for live viewing
- Docker image variant selection (
sandbox-desktop)
Minor Observations:
- The example desktop application provides a good reference implementation
- CI/cleanup workflows properly handle the new
-desktopvariant - Version synchronization between npm package and Docker images is maintained
This is a substantial but well-executed feature that enables compelling AI agent use cases. The technical implementation is sound and follows established project patterns.
Looks good to merge.
2 participants
Summary
Sandboxes can now run full desktop environments (Xvfb + XFCE4 + x11vnc + websockify) with programmatic screenshot capture. This is the container-side runtime — a Go wrapper around robotgo for native screen capture, exposed to Bun via koffi FFI, with a service/handler layer wired into the existing DI container.
Design decisions
Go FFI for screenshots rather than a CLI tool like
scrot: robotgo gives direct access to the X11 framebuffer without spawning a process per capture. The Go wrapper compiles as a c-shared library and is called from a Bun worker thread via koffi, keeping the main event loop unblocked.os.Setenv("DISPLAY", ":99")in Goinit(): In c-shared mode, Go'sos.Getenvdoesn't reliably inherit the container's environment variables. The xgb library (used internally by robotgo) callsos.Getenv("DISPLAY")at connection time, so the display must be set from Go's own environment before any capture call.koffi.disposable('HeapStr', 'str')for C string returns: koffi's default'str'return type copies the C string to a JS string but never frees the original C allocation. The disposable wrapper callsfree()after conversion, preventing a memory leak on every screenshot.robotgo
Capture()returns Go-GC-managed*image.RGBA: Unlike the lower-level C bitmap API, this doesn't require manualFreeBitmapcalls. The Go garbage collector handles cleanup.Resolution validated and clamped: Width/height must be even numbers (X11 requirement), minimum 640×480, maximum 3840×2160. The desktop manager validates at start time rather than letting Xvfb fail with a cryptic error.
This is part 1 of 4 in the desktop environment stack:
main