Skip to content

Add optional Jupyter kernel workers with private sandbox transport - #178

Draft
t-kalinowski wants to merge 9 commits into
mainfrom
feat/jupyter-kernels
Draft

t-kalinowski wants to merge 9 commits into
mainfrom
feat/jupyter-kernels

Conversation

@t-kalinowski

@t-kalinowski t-kalinowski commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Adds serve --jupyter-kernel NAME so Console can use an installed local Jupyter kernel while retaining its MCP interface and sandbox. This makes published ark and ipykernel available as optional execution backends; the built-in worker remains the default.

  • Exposes native source through send(code=...), including the synchronous and asynchronous Python clients. Kernel sessions support persistent state, text and image output, prompted input, interrupts, restart, and source archives.
  • Runs ark and ipykernel under the default read-only sandbox without manual permission changes. macOS uses Unix sockets beneath the runner's private temporary directory; Linux uses loopback inside an isolated network namespace with a deny-all proxy. Host networking remains blocked.
  • Embeds a Python adapter that owns and reaps the kernel child, correlates Jupyter replies with idle messages, preserves channel reads across startup, and keeps signed socket traffic asynchronous. The sandbox companion pin is unchanged, and no kernel source is patched or bundled.
  • Keeps exact transcript completion markers for combined control and cell calls, including Jupyter code, across split responses. A gated native startup case exercises the split while preserving its sandbox and package-preparation assertions. These collector changes affect tests only.
  • Documents setup, lifecycle and output limitations, history/export behavior, performance measurements, and external installation and binary-size costs. Website checks require the new guide in the task table and Guides sidebar.

This mode requires preinstalled packages and supports macOS/Linux. It does not provide Console's managed dependency preparation, SQL sessions, Windows worker support, or interactive notebook displays. Reticulate integration is in the dependent draft #179.

Local feature validation on macOS passed all 13 applicable ipykernel and ark cases, 22 lifecycle/CLI executions, all 36 website checks, scripts/format, and scripts/check. Seven affected baseline SQL/startup cases also passed in 12 executions. Two Linux kernel cases were capability-skipped locally. Benchmark data retains its original measured revision and is not presented as a rerun of this branch.

The hosted Unix runners capability-skip the optional Jupyter execution cases when kernel dependencies are not configured; those runs do not establish kernel execution coverage. The Windows runner reports MXC unavailable, so native sandbox execution remains unverified. Documentation deployment is intentionally skipped for pull requests.

For the collector repair, a public ipykernel restart with bundled code="pass" and timeout_ms=0 reproduced the missing completion marker and passed after the fix, followed by an accepted cell. The native R startup gate, ipykernel interrupt/restart case, and ark output/error case passed on macOS (three cases). scripts/format, git diff --check, and scripts/check passed. Current-head hosted checks remain pending; Linux and Windows were not rerun locally for this repair.

Apply the existing test-only DuckDB snapshot to direct and MXC SQL
scenarios. Assert the selected version before the long query so cold
caches cannot silently introduce the known Arrow cancellation limitation.
Preserve production defaults, interruption deadlines, and exact results.
Queue stdin while the replacement cell is held at its checkpoint, then
release it and collect the complete result. Preserve the exact output
and idle-state assertions without assuming one response completes the cell.
Require the new guide in the task table and Guides sidebar, and retain
the explicit allowlist of supported task destinations.
Keep the required completion marker when a control-and-cell response is
split across polls. Exercise the split with a gated native R startup cell
and select the same contract in related restart continuations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant