From e19cfc5d1d6d22368790183b869a31cda873827a Mon Sep 17 00:00:00 2001 From: Bobby Powers Date: Sat, 3 Oct 2026 01:10:09 -0700 Subject: [PATCH] pysimlin: run the notebook journey on a kernel that answers The journey's remaining flake was the kernel, not the page. ipykernel through 7.3 sends shell replies raw on the shell ROUTER that its ZMQStream reads; the send drains the socket's edge-triggered read edge, so a request arriving at that moment is never dispatched and the kernel sits idle for good (ipython/ipykernel#1529, first released in 7.4.0). JupyterLab sends a burst of shell requests when it connects a notebook, so the victim is a notebook's first cell: Lab sends the execute_request, the prompt shows [*], and nothing comes back. A websocket trace and a kernel stack dump of a reproduced failure show exactly that. The e2e extra requires ipykernel>=7.4. A cell run is an action with a checked effect: runCell waits for a new execution count and otherwise fails at the run, saying whether Lab sent the cell or never ran it, with the kernel state and the server log. The setup waits out a server runtime file that is still being written, and the failure artifact, which was always empty because upload-artifact skips the hidden .output directory, includes hidden files. --- .github/workflows/ci.yaml | 9 ++- src/pysimlin/CLAUDE.md | 2 +- src/pysimlin/e2e/global-setup.ts | 1 + src/pysimlin/e2e/jupyter-server.ts | 97 +++++++++++++++++++++++- src/pysimlin/e2e/kernel-session.spec.ts | 95 +++++++++++++++++++++-- src/pysimlin/e2e/notebook-editor.spec.ts | 50 +++++++++++- src/pysimlin/pyproject.toml | 9 +++ src/pysimlin/uv.lock | 52 +++++++------ 8 files changed, 278 insertions(+), 37 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index e76499405..da668bf5a 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -393,14 +393,19 @@ jobs: run: make -C src/pysimlin export-check # Screenshots, the trace, and the jupyter server log are what a failed - # journey needs to be understood. + # journey needs to be understood. They live in a dot-directory, and + # upload-artifact@v4 skips hidden files unless told otherwise (it then + # finds nothing at all). Nothing to upload is a warning, not an error: + # this step also runs when an earlier build step failed, before the + # journey wrote anything. - name: Upload journey output if: failure() uses: actions/upload-artifact@v4 with: name: pysimlin-e2e-output path: src/pysimlin/e2e/.output/ - if-no-files-found: ignore + include-hidden-files: true + if-no-files-found: warn policy: name: Policy checks diff --git a/src/pysimlin/CLAUDE.md b/src/pysimlin/CLAUDE.md index e082dfcf5..0b22d5693 100644 --- a/src/pysimlin/CLAUDE.md +++ b/src/pysimlin/CLAUDE.md @@ -16,7 +16,7 @@ For Python-specific standards (thread safety, lock ordering), see [docs/dev/pyth - `simlin/model.py` -- `Model` wrapper class; proxies `path`/`revision`/`dirty`/`save()`/`reload()` to its project, holds `selection` (written by the widget), and the display surface: `diagram()` / `_svg_mimebundle()` (static SVG), `widget()` (a fresh `ModelWidget`), and `_repr_mimebundle_` (a fresh widget's bundle merged with the SVG so static renderers show the picture). Every display creates a new widget and the model caches none of them; the imports of `simlin.widget` are lazy (`_widget_core.import_widget_module`) so `import simlin` never pays for anywidget -- which is the OPTIONAL `notebook` extra (`pip install "pysimlin[notebook]"`; the dev and e2e extras include it): without it `widget()` raises `SimlinDependencyError` (also an `ImportError`) carrying `_widget_core.install_hint()` (`%pip` under Colab) and `_repr_mimebundle_` degrades to the SVG plus a plain-text repr that carries the same message plus a `RuntimeWarning` with it, never a traceback (the warning is attributed to the user's cell by `_widget_core.user_stacklevel()`, which skips every `simlin.*` and IPython/ipykernel frame -- a bare display reaches the repr through IPython's display formatter, and a warning attributed there would name `formatters.py`, one location for every display, and be deduped for the whole kernel session; the text/plain carries the hint too because Python's once-per-location filter can still swallow a re-run of the same cell). `_widget_core.install_hint()` detects Colab exactly as anywidget does, `"google.colab.output" in sys.modules`, importing nothing. `tests/test_widget_optional.py` pins that arm by blocking the import (`sys.modules[name] = None`); every other widget test runs with the extra installed. Why an extra and why still anywidget: its chain is a couple of dozen packages that MCP/script installs never displaying a model should not carry, and a homegrown widget stack cannot replace it -- VS Code's kernel needs `import ipywidgets` and marimo needs `anywidget.AnyWidget`, so the extra is what keeps every host - `simlin/widget.py` -- `ModelWidget(anywidget.AnyWidget)`, the imperative shell of the notebook editor. Traits `project_json`/`revision` (kernel-owned, always assigned together in one `hold_sync()`), `selection` (browser-owned, mirrored to `Model.selection`), `height`/`theme`/`read_only`. Requests and replies are custom messages, never traits: browser `{type:'wasm'}` -> bytes as the first binary buffer; browser `{type:'snapshot', id, base, json}` -> `Project._apply_snapshot` -> EXACTLY ONE reply, always, and every reply (`saved`/`rejected`, from whichever arm below, including a malformed snapshot's) echoes the request's `id` verbatim (any JSON value; omitted when the request carried none) -- the browser resolves a save only on the reply naming its request, since one model displayed in several views sees every reply in every view: applied (including a `SimlinWriteError`, where the change is real and a post-commit step failed -- the write, or notifying subscribers) -> traits `(exact json received, base+1)` in one `hold_sync()` that exits BEFORE `{type:'saved'}` is sent (+ a warn `notice` naming what failed; `model.save()` is suggested only when `write_failed`); not applied (stale base, parse failure, or a handler exception BEFORE anything applied) -> no trait writes (the project's real pair is re-pushed, a no-op when unchanged), `{type:'rejected'}` + a warn `notice`; a handler exception AFTER the snapshot applied is an ACCEPT like the write-failure case -- the exact bytes received are pushed at `base+1` (best effort) and `{type:'saved'}` is sent plus a warn `notice`, because a `rejected` there costs the view a remount (undo history and any local edits since are lost) for a change that is real; a failure AFTER the reply went out (`_SnapshotProgress.replied`) sends nothing more, and a `BaseException` (KeyboardInterrupt) gets the same one reply and is then re-raised; browser `{type:'oversize', bytes}` (it refused to send a snapshot whose wire size -- the text JSON-escaped, `_widget_core.snapshot_wire_size` -- exceeds the `max_snapshot_bytes` trait, default `_widget_core.MAX_SNAPSHOT_BYTES` = 8 MiB, chosen to stay under tornado's 10 MiB `websocket_max_message_size` with only the envelope/header left to add) -> a `RuntimeWarning` on the kernel's stderr (a comm handler runs outside any cell; JupyterLab shows it in the Log Console -- the toast is what the user sees) + the same warn `notice` the browser already showed, no reply; seeding or pushing a `project_json` above the cap warns once per widget (`_warn_if_oversize`, attributed to the user's cell by `_widget_core.user_stacklevel()` -- the display, `Model.widget()` call or `edit()` block -- with IPython's display-formatter frames skipped). Subscribes to `Project.on_change` (marshalled to `shell.kernel.io_loop.add_callback` when running under ipykernel, else direct) and pushes the project's current pair plus a notice for every change that is not one of its own accepted snapshots (`_own_revisions`, a SET of the revisions its accepts produced whose notification is still queued -- ipykernel 7 handles comm messages on a subshell thread while a cell runs, so several accepts can precede the first delivery); `close()` unsubscribes. Asset delivery is resolved ONCE at import from `SIMLIN_WIDGET_ASSET` (`bundled` | `inline` | http(s) URL) into `_ASSETS`; a missing asset never breaks `import simlin` -- creating a widget raises `SimlinAssetError`. The `inline` shim contract with the JS (`globalThis.__simlinWidgetInlineWasm = "";` on the line before the module) is documented on `_widget_core.inline_esm` - `simlin/_widget_core.py` -- the widget's pure decisions: `parse_asset_mode`/`inline_esm`/`missing_asset_message`, `parse_incoming` (message classification; malformed messages are `Unrecognised`, never exceptions), `plan_snapshot_reply` (accept / stale / raised arms), the snapshot size guard (`MAX_SNAPSHOT_BYTES` with its rationale, `snapshot_wire_size`, `oversize_warning`/`oversize_report_warning`/`oversize_notice`, `format_size` -- mirrored byte for byte by `snapshotWireSize`/`formatSize` in the JS, with one fixture list pinned in both suites), `is_own_change`, `notice_for_change` (one text per `ChangeSource`), `dispatch_for_shell`. `tests/test_widget_core.py` enumerates every arm; `tests/test_widget.py` drives the shell through a recording comm (no browser, no kernel) -- `e2e/` -- the JupyterLab journey (Playwright; design AC4.2): `global-setup.ts` launches `jupyter lab` from this venv (`jupyter-server.ts`: OS-assigned port, random token, temporary config/data/runtime/root directories, a preflight that the interpreter imports `simlin` with staged widget assets and `jupyterlab`) and hands the URL/token/root to the spec through `SIMLIN_E2E_*` env vars; each notebook's kernel session is started on the server before the browser opens the notebook (`startNotebookSession`, tried `SESSION_START_ATTEMPTS` times), because Lab connects to a session that exists and leaves a notebook kernel-less for good when its own one start request fails; a cell is run only once Lab's own session context says its kernel is connected (`kernelState`, read from `window.jupyterapp`, which the server is launched to expose -- the toolbar's execution indicator reads `idle` before the session has connected to anything, and a run in that window races Lab's session start); each notebook opens in a JupyterLab workspace of its own and is confirmed by path before anything is clicked (`openNotebook`), because Lab restores a shared workspace's earlier notebooks before it opens the one the URL names, and a helper that means "the visible notebook" then runs a cell of the wrong one; `kernel-session.spec.ts` pins the session-start and kernel-state decisions without a browser; `notebook-editor.spec.ts` drives the real JupyterLab UI: `simlin.open` + display -> the Editor renders; a variable added and given an equation in the Editor -> the file on disk changes and a following `m.run()` cell reports it; `m.selection` names the selected variable; a Python `edit()` -> "Updated from Python" in the Editor; a second-process write -> "Updated on disk" and `m.revision` advanced, and after those remounts `m.selection` is `()` again. It is NOT part of `pytest`/pre-commit: `make e2e` runs it (CI: the `pysimlin-e2e` job); it needs the `e2e` extra (jupyterlab, in THIS venv so its share/jupyter carries the anywidget/ipywidgets labextensions), the CFFI extension, staged `simlin/_widget/` assets (`make assets`), and Playwright's chromium. `SIMLIN_E2E_PYTHON` picks another interpreter; `SIMLIN_E2E_KEEP_TMP` keeps the server tree; the server log (jupyter-lab.log) is written to the git-ignored output directory beside the spec (see `e2e/playwright.config.ts`) +- `e2e/` -- the JupyterLab journey (Playwright; design AC4.2): `global-setup.ts` launches `jupyter lab` from this venv (`jupyter-server.ts`: OS-assigned port, random token, temporary config/data/runtime/root directories, a preflight that the interpreter imports `simlin` with staged widget assets and `jupyterlab`, and the runtime file read only once whole -- `readServerInfo`, since the server writes it in place) and hands the URL/token/root/server-log path to the spec through `SIMLIN_E2E_*` env vars; each notebook's kernel session is started on the server before the browser opens the notebook (`startNotebookSession`, tried `SESSION_START_ATTEMPTS` times), because Lab connects to a session that exists and leaves a notebook kernel-less for good when its own one start request fails; a cell is run only once Lab's own session context says its kernel is connected (`kernelState`, read from `window.jupyterapp`, which the server is launched to expose -- the toolbar's execution indicator reads `idle` before the session has connected to anything, and a run in that window races Lab's session start); a cell run is an action with a checked effect: `runCell` waits for the prompt to show a new execution count (`parsePrompt`/`cellRan`) and otherwise fails at the run with `unrunCellReport` (`[*]` = sent and unanswered by the kernel, no new count = Lab never ran it; the kernel's state and the server log's tail) rather than as an output that never appears; the `e2e` extra requires ipykernel >= 7.4, the first release that sends shell replies through the shell socket's ZMQStream (ipython/ipykernel#1529) -- before it a reply sent raw on that socket could strand a request arriving at the same moment, Lab's burst of requests when it connects a notebook made that likely on a notebook's first cell, and the kernel then sat idle with the cell at `[*]` for good; each notebook opens in a JupyterLab workspace of its own and is confirmed by path before anything is clicked (`openNotebook`), because Lab restores a shared workspace's earlier notebooks before it opens the one the URL names, and a helper that means "the visible notebook" then runs a cell of the wrong one; `kernel-session.spec.ts` pins the runtime-file, session-start, kernel-state, and cell-run decisions without a browser; `notebook-editor.spec.ts` drives the real JupyterLab UI: `simlin.open` + display -> the Editor renders; a variable added and given an equation in the Editor -> the file on disk changes and a following `m.run()` cell reports it; `m.selection` names the selected variable; a Python `edit()` -> "Updated from Python" in the Editor; a second-process write -> "Updated on disk" and `m.revision` advanced, and after those remounts `m.selection` is `()` again. It is NOT part of `pytest`/pre-commit: `make e2e` runs it (CI: the `pysimlin-e2e` job); it needs the `e2e` extra (jupyterlab, in THIS venv so its share/jupyter carries the anywidget/ipywidgets labextensions), the CFFI extension, staged `simlin/_widget/` assets (`make assets`), and Playwright's chromium. `SIMLIN_E2E_PYTHON` picks another interpreter; `SIMLIN_E2E_KEEP_TMP` keeps the server tree; the server log (jupyter-lab.log) is written to the git-ignored output directory beside the spec (see `e2e/playwright.config.ts`) - `scripts/check_notebook_export.py` -- executes an example notebook headless with nbconvert and checks both static-export arms of the display cell: with the stored widget state the HTML carries the widget view plus the embedded state (the `_esm` rides along), and with `metadata.widgets` stripped -- what GitHub or a notebook saved without state gives a static renderer -- nbconvert's `WidgetsDataTypeFilter` skips the widget-view mimetype and the HTML carries the SVG (the lab template embeds it as a base64 ``). Its pure half is unit-tested in `tests/test_check_notebook_export.py`; the shell runs under `make export-check` and the CI `pysimlin-e2e` job, never as a pytest (the dev extra has no nbconvert and a skipped test proves nothing) - `docs/notebook-hosts.md` -- per-host checklist for the notebook editor (JupyterLab 4, Notebook 7, VS Code local/Remote-SSH, Colab, marimo) with each host's VERIFIED/UNVERIFIED status; only JupyterLab is verified (by `e2e/`). Update the status line in the same PR that runs a checklist. `examples/colab_quickstart.ipynb` is the Colab entry point (its `%pip` cell is skipped by `tests/test_examples.py`'s script run) - `simlin/_widget/` -- package data: `widget.js` and `libsimlin-browser.wasm`, build outputs of the `@simlin/notebook-widget` TypeScript package (git-ignored; see its README.md and `pyproject.toml` package-data) diff --git a/src/pysimlin/e2e/global-setup.ts b/src/pysimlin/e2e/global-setup.ts index fe1a10620..ea3d15421 100644 --- a/src/pysimlin/e2e/global-setup.ts +++ b/src/pysimlin/e2e/global-setup.ts @@ -19,4 +19,5 @@ export default async function globalSetup(): Promise { process.env[ENV.url] = server.url; process.env[ENV.token] = server.token; process.env[ENV.rootDir] = server.rootDir; + process.env[ENV.logPath] = server.logPath; } diff --git a/src/pysimlin/e2e/jupyter-server.ts b/src/pysimlin/e2e/jupyter-server.ts index ba09aa5be..198bdb2ad 100644 --- a/src/pysimlin/e2e/jupyter-server.ts +++ b/src/pysimlin/e2e/jupyter-server.ts @@ -38,6 +38,7 @@ export const ENV = { url: 'SIMLIN_E2E_JUPYTER_URL', token: 'SIMLIN_E2E_JUPYTER_TOKEN', rootDir: 'SIMLIN_E2E_ROOT_DIR', + logPath: 'SIMLIN_E2E_SERVER_LOG', } as const; const here = __dirname; @@ -93,7 +94,28 @@ export function userSettingsFiles(configDir: string): Array<{ file: string; cont ]; } -/** Read `url`/`token`/`root_dir` from a `jpserver-.json` runtime file. */ +/** + * Read `url`/`token`/`root_dir` from a `jpserver-.json` runtime file, + * or `undefined` while the file is still being written. + * + * The server writes the file in place (jupyter_server's `secure_write` opens + * the final path and `json.dump`s into it), so a reader that finds the file + * can read it truncated, or empty. Text that does not parse yet is that + * window; a file that parses but lacks a field is a real error. + */ +export function readServerInfo(json: string): ServerInfo | undefined { + try { + JSON.parse(json); + } catch (err) { + if (err instanceof SyntaxError) { + return undefined; + } + throw err; + } + return parseServerInfo(json); +} + +/** Read `url`/`token`/`root_dir` from a complete `jpserver-.json` runtime file. */ export function parseServerInfo(json: string): ServerInfo { const info = JSON.parse(json) as { url?: unknown; token?: unknown; root_dir?: unknown }; if (typeof info.url !== 'string' || typeof info.token !== 'string' || typeof info.root_dir !== 'string') { @@ -266,6 +288,77 @@ export function canRunCells(state: string): boolean { return state === 'idle' || state === 'busy'; } +/** + * A code cell's execution as its input prompt shows it: `null` for never run + * (`[ ]:`), `'running'` for sent to the kernel and not yet answered (`[*]:`), + * or the execution count the kernel replied with (`[3]:`). + */ +export type CellExecution = null | 'running' | number; + +/** Read a cell's input prompt text (`[ ]:`, `[*]:`, `[3]:`). */ +export function parsePrompt(text: string): CellExecution { + const m = /^\s*\[\s*([^\]]*?)\s*\]:?\s*$/.exec(text); + if (m === null) { + throw new Error(`not a cell input prompt: ${JSON.stringify(text)}`); + } + if (m[1] === '') { + return null; + } + if (m[1] === '*') { + return 'running'; + } + if (/^\d+$/.test(m[1])) { + return Number(m[1]); + } + throw new Error(`not a cell input prompt: ${JSON.stringify(text)}`); +} + +/** + * Whether a run of a cell whose prompt read `before` has been answered by the + * kernel, now that it reads `now`: the kernel's reply carries a new execution + * count, so a count that differs from the one the cell had is the kernel's + * word that it ran the cell. A cell that shows no count, or `[*]`, or the + * count it had before, has not been run by this press. + */ +export function cellRan(before: CellExecution, now: CellExecution): boolean { + return typeof now === 'number' && now !== before; +} + +/** + * Why a cell run the journey asked for never came back, in terms that say + * which side to look at: a `[*]` prompt means JupyterLab sent the request and + * the kernel has not answered (look at the kernel and the server log), and a + * prompt with no new count means JupyterLab never ran the cell at all (look at + * the page: focus, a dialog, a cancelled run). + */ +export function unrunCellReport(facts: { + notebook: string; + index: number; + before: CellExecution; + now: CellExecution; + kernel: string; + waitedMs: number; + serverLogTail: string; +}): string { + const shown = (e: CellExecution): string => (e === null ? '[ ]' : e === 'running' ? '[*]' : `[${e}]`); + const verdict = + facts.now === 'running' + ? 'JupyterLab sent it to the kernel, which has not answered' + : 'JupyterLab never ran it (no execution request went out, or the run was cancelled)'; + return [ + `cell ${facts.index} of ${facts.notebook} did not run within ${facts.waitedMs} ms: ${verdict}.`, + `prompt before the run: ${shown(facts.before)}; now: ${shown(facts.now)}; notebook kernel: ${facts.kernel}`, + `last lines of the jupyter server log:`, + facts.serverLogTail, + ].join('\n'); +} + +/** The last `lines` lines of `text`, for an error message. */ +export function tail(text: string, lines: number): string { + const all = text.trimEnd().split('\n'); + return all.slice(Math.max(0, all.length - lines)).join('\n'); +} + /** Launch JupyterLab and resolve once its REST API answers with our token. */ export async function launchJupyterLab(): Promise { const python = pythonExecutable(); @@ -322,7 +415,7 @@ export async function launchJupyterLab(): Promise { if (files.length === 0) { return undefined; } - return parseServerInfo(fs.readFileSync(path.join(runtimeDir, files[0]), 'utf8')); + return readServerInfo(fs.readFileSync(path.join(runtimeDir, files[0]), 'utf8')); }); const url = info.url; const bearer = info.token; diff --git a/src/pysimlin/e2e/kernel-session.spec.ts b/src/pysimlin/e2e/kernel-session.spec.ts index b087c5462..3d8055419 100644 --- a/src/pysimlin/e2e/kernel-session.spec.ts +++ b/src/pysimlin/e2e/kernel-session.spec.ts @@ -3,12 +3,16 @@ // Version 2.0, that can be found in the LICENSE file. /** - * The two decisions the journey makes about a notebook's kernel, apart from - * any browser: how its session is started (`startNotebookSession`: one - * request per attempt, a refusal tried again, a server that refuses every - * time reported with each refusal's own words) and when a cell run would - * execute (`kernelState`, `canRunCells`). The server here is a stand-in for - * `fetch` and the session facts are written out; the journey itself exercises + * The decisions the journey makes about its server and a notebook's kernel, + * apart from any browser: when the server's runtime file can be read + * (`readServerInfo`: a file still being written is waited out, a complete one + * missing a field is an error); how a notebook's session is started + * (`startNotebookSession`: one request per attempt, a refusal tried again, a + * server that refuses every time reported with each refusal's own words); + * when a cell run would execute (`kernelState`, `canRunCells`); and whether a + * run was answered, and what to say when it was not (`parsePrompt`, + * `cellRan`, `unrunCellReport`). The server here is a stand-in for `fetch` and + * the session facts and prompts are written out; the journey itself exercises * the real ones. */ @@ -18,9 +22,15 @@ import { KERNEL_NAME, SESSION_START_ATTEMPTS, canRunCells, + cellRan, kernelState, + parsePrompt, + readServerInfo, sessionRequestBody, startNotebookSession, + tail, + unrunCellReport, + type CellExecution, type SessionFacts, } from './jupyter-server'; @@ -118,3 +128,76 @@ test('a cell run executes only in a started session with a connected, live kerne expect(canRunCells(state), state).toBe(runs); } }); + +const SERVER_INFO = JSON.stringify({ url: 'http://127.0.0.1:8888/', token: 't', root_dir: '/r', pid: 1 }, null, 2); + +test('the runtime file is read once it is whole, and waited for while it is being written', () => { + expect(readServerInfo(SERVER_INFO)).toEqual({ url: 'http://127.0.0.1:8888/', token: 't', rootDir: '/r' }); + // The server writes the file in place: an empty file, and every truncation + // of a whole one, is the write still under way. + for (let n = 0; n < SERVER_INFO.length; n++) { + expect(readServerInfo(SERVER_INFO.slice(0, n)), `first ${n} characters`).toBeUndefined(); + } + // Whole but without a field the journey needs: no amount of waiting fixes it. + expect(() => readServerInfo(JSON.stringify({ url: 'http://127.0.0.1:8888/', token: 't' }))).toThrow( + /unexpected jupyter server info/, + ); +}); + +// Every prompt a code cell shows, as JupyterLab renders it (the empty prompt +// carries a non-breaking space). +const PROMPTS: Array<{ text: string; execution: CellExecution }> = [ + { text: '[\u00a0]:', execution: null }, + { text: '[ ]:', execution: null }, + { text: '[*]:', execution: 'running' }, + { text: '[1]:', execution: 1 }, + { text: '[42]:', execution: 42 }, +]; + +test('a cell prompt reads as never run, running, or the execution count', () => { + for (const { text, execution } of PROMPTS) { + expect(parsePrompt(text), JSON.stringify(text)).toBe(execution); + } + for (const text of ['', 'In [1]:', '[x]:', '[1']) { + expect(() => parsePrompt(text), JSON.stringify(text)).toThrow(/not a cell input prompt/); + } +}); + +// Every (before, now) pair of execution states: a run is answered only when +// the prompt shows a count it did not show before. +const EXECUTIONS: CellExecution[] = [null, 'running', 3, 4]; + +test('a run is answered exactly when the prompt shows a new execution count', () => { + for (const before of EXECUTIONS) { + for (const now of EXECUTIONS) { + const answered = typeof now === 'number' && now !== before; + expect(cellRan(before, now), `${String(before)} -> ${String(now)}`).toBe(answered); + } + } +}); + +test('an unanswered run says which side never acted, with the prompt, the kernel, and the server log', () => { + const facts = { + notebook: 'scratch.ipynb', + index: 0, + before: null, + kernel: 'idle', + waitedMs: 60_000, + serverLogTail: '[I ServerApp] Connecting to kernel k.', + }; + const sent = unrunCellReport({ ...facts, now: 'running' }); + expect(sent).toContain('cell 0 of scratch.ipynb did not run within 60000 ms'); + expect(sent).toContain('JupyterLab sent it to the kernel, which has not answered'); + expect(sent).toContain('prompt before the run: [ ]; now: [*]; notebook kernel: idle'); + expect(sent).toContain('[I ServerApp] Connecting to kernel k.'); + for (const now of [null, 3] as CellExecution[]) { + const unsent = unrunCellReport({ ...facts, before: 3, now }); + expect(unsent).toContain('JupyterLab never ran it'); + expect(unsent).toContain(`prompt before the run: [3]; now: ${now === null ? '[ ]' : '[3]'}`); + } +}); + +test('a log tail keeps the last lines', () => { + expect(tail('a\nb\nc\n', 2)).toBe('b\nc'); + expect(tail('a\nb', 5)).toBe('a\nb'); +}); diff --git a/src/pysimlin/e2e/notebook-editor.spec.ts b/src/pysimlin/e2e/notebook-editor.spec.ts index 6edc7d1df..12284ac8f 100644 --- a/src/pysimlin/e2e/notebook-editor.spec.ts +++ b/src/pysimlin/e2e/notebook-editor.spec.ts @@ -33,10 +33,14 @@ import { test, expect, type Locator, type Page } from '@playwright/test'; import { ENV, canRunCells, + cellRan, kernelState, + parsePrompt, pysimlinDir, pythonExecutable, startNotebookSession, + tail, + unrunCellReport, type SessionFacts, } from './jupyter-server'; @@ -252,12 +256,56 @@ async function waitForKernel(page: Page): Promise { } } -/** Click into cell `index`'s editor and run it with Shift+Enter. */ +/** How long a cell run may take to be answered; the journey's cells take seconds. */ +const CELL_RUN_TIMEOUT_MS = 60_000; + +/** + * Click into cell `index`'s editor, run it with Shift+Enter, and wait until + * the kernel has answered the run (`cellRan`: the prompt shows a new + * execution count). + * + * A run that is never answered otherwise surfaces later as an output that + * never appears, which says nothing about why; this fails at the run instead, + * with what the page and the server know (`unrunCellReport`). + */ async function runCell(page: Page, index: number): Promise { await waitForKernel(page); const cell = activeCells(page).nth(index); + const prompt = cell.locator('.jp-InputPrompt'); + const before = parsePrompt((await prompt.textContent()) ?? ''); await cell.locator('.jp-InputArea-editor').click(); await page.keyboard.press('Shift+Enter'); + let now = before; + try { + await expect + .poll( + async () => { + now = parsePrompt((await prompt.textContent()) ?? ''); + return cellRan(before, now); + }, + { timeout: CELL_RUN_TIMEOUT_MS }, + ) + .toBe(true); + } catch { + const notebook = (await visibleNotebookPath(page)) ?? 'the visible notebook'; + let serverLog: string; + try { + serverLog = tail(fs.readFileSync(required(ENV.logPath), 'utf8'), 20); + } catch (err) { + serverLog = `(unreadable: ${err instanceof Error ? err.message : String(err)})`; + } + throw new Error( + unrunCellReport({ + notebook, + index, + before, + now, + kernel: kernelState(await sessionFacts(page)), + waitedMs: CELL_RUN_TIMEOUT_MS, + serverLogTail: serverLog, + }), + ); + } return cell; } diff --git a/src/pysimlin/pyproject.toml b/src/pysimlin/pyproject.toml index 173595bb8..2e6fcf965 100644 --- a/src/pysimlin/pyproject.toml +++ b/src/pysimlin/pyproject.toml @@ -96,9 +96,18 @@ notebooks = [ # never import. It must live in THIS venv (not `uv run --with jupyterlab`, # whose ephemeral prefix hides the anywidget/ipywidgets labextensions that # ship under this venv's share/jupyter). +# +# ipykernel 7.4 is the first release that sends shell replies through the +# shell socket's ZMQStream (ipython/ipykernel#1529). Before it, a reply sent +# raw on that socket could consume the edge of a request arriving at the same +# moment, and the kernel then sat idle with the request unread, for good. +# Lab's burst of requests when it connects a notebook (history, debug, the +# widget manager's comm) makes that race likely on a notebook's first cell: +# the cell shows [*] and never runs. e2e = [ "pysimlin[notebook]", "jupyterlab>=4.3", + "ipykernel>=7.4", ] [project.urls] diff --git a/src/pysimlin/uv.lock b/src/pysimlin/uv.lock index 5e6aa5db9..7ead11f25 100644 --- a/src/pysimlin/uv.lock +++ b/src/pysimlin/uv.lock @@ -816,7 +816,7 @@ wheels = [ [[package]] name = "ipykernel" -version = "7.2.0" +version = "7.4.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "appnope", marker = "sys_platform == 'darwin'" }, @@ -826,16 +826,15 @@ dependencies = [ { name = "jupyter-client" }, { name = "jupyter-core" }, { name = "matplotlib-inline" }, - { name = "nest-asyncio" }, + { name = "nest-asyncio2" }, { name = "packaging" }, - { name = "psutil" }, { name = "pyzmq" }, { name = "tornado" }, { name = "traitlets" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ca/8d/b68b728e2d06b9e0051019640a40a9eb7a88fcd82c2e1b5ce70bef5ff044/ipykernel-7.2.0.tar.gz", hash = "sha256:18ed160b6dee2cbb16e5f3575858bc19d8f1fe6046a9a680c708494ce31d909e", size = 176046, upload-time = "2026-02-06T16:43:27.403Z" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/05/e5499c8b762387d83926bf0389e8fff888c9bf6e7af30566a8527bc12f4a/ipykernel-7.4.0.tar.gz", hash = "sha256:4330114d22b9b33575b2c7c68753fc8faabbbcfbc60b0cdff9f14dd4ce63742f", size = 192731, upload-time = "2026-09-29T16:33:11.468Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/82/b9/e73d5d9f405cba7706c539aa8b311b49d4c2f3d698d9c12f815231169c71/ipykernel-7.2.0-py3-none-any.whl", hash = "sha256:3bbd4420d2b3cc105cbdf3756bfc04500b1e52f090a90716851f3916c62e1661", size = 118788, upload-time = "2026-02-06T16:43:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/a5/f8/8b5c10c7f36ab5a3b5c4703669e5b14e44e98782b58e535dd30670e8799a/ipykernel-7.4.0-py3-none-any.whl", hash = "sha256:a6757f790ddc5a6006b813d82da1a9dbb65b3a00f2b6607df776418c6566ec1c", size = 123184, upload-time = "2026-09-29T16:33:09.612Z" }, ] [[package]] @@ -998,7 +997,7 @@ wheels = [ [[package]] name = "jupyter-client" -version = "8.8.0" +version = "8.10.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jupyter-core" }, @@ -1006,10 +1005,11 @@ dependencies = [ { name = "pyzmq" }, { name = "tornado" }, { name = "traitlets" }, + { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/05/e4/ba649102a3bc3fbca54e7239fb924fd434c766f855693d86de0b1f2bec81/jupyter_client-8.8.0.tar.gz", hash = "sha256:d556811419a4f2d96c869af34e854e3f059b7cc2d6d01a9cd9c85c267691be3e", size = 348020, upload-time = "2026-01-08T13:55:47.938Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c5/2a/906772148a06e48885039e0250c340b770a55bbf37b08d6ee0449df369c3/jupyter_client-8.10.0.tar.gz", hash = "sha256:9f7116294dca55f1785be880057d44544db9b1567718d92cb33c58886afb9497", size = 360653, upload-time = "2026-08-28T12:17:10.854Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2d/0b/ceb7694d864abc0a047649aec263878acb9f792e1fec3e676f22dc9015e3/jupyter_client-8.8.0-py3-none-any.whl", hash = "sha256:f93a5b99c5e23a507b773d3a1136bd6e16c67883ccdbd9a829b0bbdb98cd7d7a", size = 107371, upload-time = "2026-01-08T13:55:45.562Z" }, + { url = "https://files.pythonhosted.org/packages/0f/88/7c548de1f6c2ade7c931a3282da73f9274fa6a1531091be682f89c85efb9/jupyter_client-8.10.0-py3-none-any.whl", hash = "sha256:5f73f24f22fa25192cfff6b23c051932a2473a797b05734aff495b392103e14e", size = 110184, upload-time = "2026-08-28T12:17:09.028Z" }, ] [[package]] @@ -1601,12 +1601,12 @@ wheels = [ ] [[package]] -name = "nest-asyncio" -version = "1.6.0" +name = "nest-asyncio2" +version = "1.7.3" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" } +sdist = { url = "https://files.pythonhosted.org/packages/5e/a6/a2775b388a14b5ea0c894bbe513c0d3e04e65e8ae583e60f5357722b8e03/nest_asyncio2-1.7.3.tar.gz", hash = "sha256:2e9a84d5d1efe6d020c72988d21aec569bac42d98af2ff6b9de24640c5d22a34", size = 15642, upload-time = "2026-09-22T17:15:06.709Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" }, + { url = "https://files.pythonhosted.org/packages/29/dd/a8bebc78975a7466932077a0973cbab4361a32368b1d28d40683ec19333c/nest_asyncio2-1.7.3-py3-none-any.whl", hash = "sha256:2bc87bdca654e719425145f5e84eaeb0080b013fdd47d65729a7d66243f4987c", size = 8364, upload-time = "2026-09-22T17:15:05.492Z" }, ] [[package]] @@ -2089,6 +2089,7 @@ dev = [ ] e2e = [ { name = "anywidget" }, + { name = "ipykernel" }, { name = "jupyterlab" }, ] notebook = [ @@ -2117,6 +2118,7 @@ requires-dist = [ { name = "cattrs", specifier = ">=23" }, { name = "cffi", specifier = ">=1.15.0" }, { name = "hypothesis", marker = "extra == 'test'", specifier = ">=6.100.0" }, + { name = "ipykernel", marker = "extra == 'e2e'", specifier = ">=7.4" }, { name = "ipykernel", marker = "extra == 'notebooks'", specifier = ">=6.29" }, { name = "jsonschema", marker = "extra == 'test'", specifier = ">=4.20.0" }, { name = "jupyterlab", marker = "extra == 'e2e'", specifier = ">=4.3" }, @@ -2685,19 +2687,19 @@ wheels = [ [[package]] name = "tornado" -version = "6.5.6" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/50/57/6d7303a77ae439d9189108f76c0c4fd89ee5e2cc8387bffb55232565c4ed/tornado-6.5.6.tar.gz", hash = "sha256:9a365179fe8ff6b8766f602c0f67c185d778193e9bdd828b19f0b6ed7764177d", size = 518139, upload-time = "2026-05-27T15:35:54.646Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/1b/0d/b4f481e18c5a51864e6d12b9a05ecf72919696680b747c958c3fc1f4fbae/tornado-6.5.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:65fcfaafb079435c2c19dc9e07c0f1cf0fa9051759ed0a7d0a3ba7ea7f64919c", size = 447737, upload-time = "2026-05-27T15:35:38.122Z" }, - { url = "https://files.pythonhosted.org/packages/9e/9c/5430c39fcab1144d35860f457b15e9c08b4bc7ac86764354204e983d6183/tornado-6.5.6-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:38bc01b4acacded2de63ae78023548e41ebe6fbed3ec05a796d7ae3ad893887e", size = 445899, upload-time = "2026-05-27T15:35:40.519Z" }, - { url = "https://files.pythonhosted.org/packages/8b/79/fa7e14a2f939c807a8d30619b4eb604eab219601b78792516ebe22d40cf9/tornado-6.5.6-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b942e6a137fda31ff54bf8e6e2c8d1c37f1f50583f3ed53fb840b53b9601d104", size = 448964, upload-time = "2026-05-27T15:35:42.106Z" }, - { url = "https://files.pythonhosted.org/packages/a7/71/bd67d5f5199f937dafe03a49a37989f60f600ff6fef34c79412a829d97bd/tornado-6.5.6-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8666946e70171b8c3f1fc9b7876fac492e84822c4c7f3746f4e8f8bc9ac92a79", size = 449935, upload-time = "2026-05-27T15:35:43.906Z" }, - { url = "https://files.pythonhosted.org/packages/cc/a4/c24388c9cf5b3c3a513b56a158af9f23092c9a2810d789e294310797df21/tornado-6.5.6-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1c34cfab7ad6d104f052f55de06d39bbafc5885cfeb4da688803308dbcfa90b7", size = 449767, upload-time = "2026-05-27T15:35:45.793Z" }, - { url = "https://files.pythonhosted.org/packages/a5/eb/6a07ad550c3f7b37244bd0becdf293ec3d3e961783d8b720a97df50de1b2/tornado-6.5.6-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:385f35e4e22fb52551dfcda4cdc8c30c61c2c001aef5ddad99cdfe116952efd3", size = 449174, upload-time = "2026-05-27T15:35:47.485Z" }, - { url = "https://files.pythonhosted.org/packages/bb/84/3469e098dccdb6763130e06aacd786bb4363fca7b590a55c101ddf34ed30/tornado-6.5.6-cp39-abi3-win32.whl", hash = "sha256:db475f1b67b2809b10bb16264829087724ca8d24fe4ed47f7b8675cae453ef86", size = 450230, upload-time = "2026-05-27T15:35:49.322Z" }, - { url = "https://files.pythonhosted.org/packages/d2/3c/273a04e0b9dd9016f1685cca0c1c8795a71ac88a34a8c889a0b443483226/tornado-6.5.6-cp39-abi3-win_amd64.whl", hash = "sha256:6739bf1e8eb09230f1280ddbd3236f0309db70f2c551a8dbc40f62babdf82f79", size = 450667, upload-time = "2026-05-27T15:35:51.194Z" }, - { url = "https://files.pythonhosted.org/packages/02/98/0cffe22a224f60c5fb1e3aa0b76f9da2e1ca78b0e9545e3d077c68ce60a7/tornado-6.5.6-cp39-abi3-win_arm64.whl", hash = "sha256:2543597b24a695d72338a9a77818362d72387c03ae173f1f169eadc5c91466ac", size = 449690, upload-time = "2026-05-27T15:35:52.902Z" }, +version = "6.5.10" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/06/61/53d562a57b28c08eda40b258c0f975e360541943ad7c7bef897a40caafda/tornado-6.5.10.tar.gz", hash = "sha256:a6b1ccd08c04b4a06fb5aeb381be99de5ad1e5375c1785e31d78c880feb57687", size = 537910, upload-time = "2026-09-15T13:47:48.73Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cd/5b/ff5fc58fa2427c30dea74c90053f4fc5eda1e7f3833ed3ecc7147fe2b311/tornado-6.5.10-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9261783640e23258694a9ff0795df430a5a7b0a651d3dd53dd0969ad6be16da7", size = 465883, upload-time = "2026-09-15T13:47:35.463Z" }, + { url = "https://files.pythonhosted.org/packages/ad/f5/cd7be26c34a3315532f3aef5f092465da8f59c334dd439d3c14aaef16461/tornado-6.5.10-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:83e6cf438b106c6b3852d70960967bb1b70c87438050dca0981e4b9aa751a4c1", size = 464046, upload-time = "2026-09-15T13:47:37.178Z" }, + { url = "https://files.pythonhosted.org/packages/60/33/df6d7d04854a58619f8349a51e3edb138324130a7562b0bb21f115bb940f/tornado-6.5.10-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bdf942448169e5336451d0494d7e3d81cfa726d5aa312affdc4682dd62a62f6d", size = 467096, upload-time = "2026-09-15T13:47:38.559Z" }, + { url = "https://files.pythonhosted.org/packages/29/17/cc35dff68272d685cffd8600ffafbd8067e7d05e7348d9f80caddffbbd5f/tornado-6.5.10-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:69acca6501eed74582b76dbbceee2a91613f54728e3e418346000d7103101676", size = 468067, upload-time = "2026-09-15T13:47:40.085Z" }, + { url = "https://files.pythonhosted.org/packages/c3/01/6e5349b4e1a53a4b4972a6716785e1fe7407f312063c3972690af8ff301b/tornado-6.5.10-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:66aaa3f57d30c6e6becee83ff28055d5930ac724214bde99393eefda83d5e015", size = 467901, upload-time = "2026-09-15T13:47:41.576Z" }, + { url = "https://files.pythonhosted.org/packages/28/5e/b4facf94370dba006819c8d304376f8b9fbec6b935b5e51bf45823a9790b/tornado-6.5.10-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4bd192b959f9128fb99b8898148070ba4574c9589b78bce42d1851131fe85828", size = 467308, upload-time = "2026-09-15T13:47:43.145Z" }, + { url = "https://files.pythonhosted.org/packages/56/ae/047938e828cafc8eca4c908fafb6588fee944e3af39a0af9d7b602499ae5/tornado-6.5.10-cp39-abi3-win32.whl", hash = "sha256:302eb1e0e3e159314eb591920529fdea80acca92df5510a2cec5bbd4f099ec72", size = 468387, upload-time = "2026-09-15T13:47:44.556Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d4/5901517f05affd752490f6a654ba31b7474664e8dd80bd045a00c220bd88/tornado-6.5.10-cp39-abi3-win_amd64.whl", hash = "sha256:37ae8f150cecfdbf747fc4e12f5e9a97ecd8cf1d4cdb3f119e2de84b11196918", size = 468828, upload-time = "2026-09-15T13:47:45.961Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1a/fd497f3a7f7b74bb04f4b94536b5c9f80742b5d50501fd27977652ddec16/tornado-6.5.10-cp39-abi3-win_arm64.whl", hash = "sha256:ce045d3c298fddd30e89a2777f97039d1b641eb9518ac7b26a4721903539c694", size = 467847, upload-time = "2026-09-15T13:47:47.283Z" }, ] [[package]]