diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index e76499405..da668bf5a 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -393,14 +393,19 @@ jobs: run: make -C src/pysimlin export-check # Screenshots, the trace, and the jupyter server log are what a failed - # journey needs to be understood. + # journey needs to be understood. They live in a dot-directory, and + # upload-artifact@v4 skips hidden files unless told otherwise (it then + # finds nothing at all). Nothing to upload is a warning, not an error: + # this step also runs when an earlier build step failed, before the + # journey wrote anything. - name: Upload journey output if: failure() uses: actions/upload-artifact@v4 with: name: pysimlin-e2e-output path: src/pysimlin/e2e/.output/ - if-no-files-found: ignore + include-hidden-files: true + if-no-files-found: warn policy: name: Policy checks diff --git a/src/pysimlin/CLAUDE.md b/src/pysimlin/CLAUDE.md index e082dfcf5..0b22d5693 100644 --- a/src/pysimlin/CLAUDE.md +++ b/src/pysimlin/CLAUDE.md @@ -16,7 +16,7 @@ For Python-specific standards (thread safety, lock ordering), see [docs/dev/pyth - `simlin/model.py` -- `Model` wrapper class; proxies `path`/`revision`/`dirty`/`save()`/`reload()` to its project, holds `selection` (written by the widget), and the display surface: `diagram()` / `_svg_mimebundle()` (static SVG), `widget()` (a fresh `ModelWidget`), and `_repr_mimebundle_` (a fresh widget's bundle merged with the SVG so static renderers show the picture). Every display creates a new widget and the model caches none of them; the imports of `simlin.widget` are lazy (`_widget_core.import_widget_module`) so `import simlin` never pays for anywidget -- which is the OPTIONAL `notebook` extra (`pip install "pysimlin[notebook]"`; the dev and e2e extras include it): without it `widget()` raises `SimlinDependencyError` (also an `ImportError`) carrying `_widget_core.install_hint()` (`%pip` under Colab) and `_repr_mimebundle_` degrades to the SVG plus a plain-text repr that carries the same message plus a `RuntimeWarning` with it, never a traceback (the warning is attributed to the user's cell by `_widget_core.user_stacklevel()`, which skips every `simlin.*` and IPython/ipykernel frame -- a bare display reaches the repr through IPython's display formatter, and a warning attributed there would name `formatters.py`, one location for every display, and be deduped for the whole kernel session; the text/plain carries the hint too because Python's once-per-location filter can still swallow a re-run of the same cell). `_widget_core.install_hint()` detects Colab exactly as anywidget does, `"google.colab.output" in sys.modules`, importing nothing. `tests/test_widget_optional.py` pins that arm by blocking the import (`sys.modules[name] = None`); every other widget test runs with the extra installed. Why an extra and why still anywidget: its chain is a couple of dozen packages that MCP/script installs never displaying a model should not carry, and a homegrown widget stack cannot replace it -- VS Code's kernel needs `import ipywidgets` and marimo needs `anywidget.AnyWidget`, so the extra is what keeps every host - `simlin/widget.py` -- `ModelWidget(anywidget.AnyWidget)`, the imperative shell of the notebook editor. Traits `project_json`/`revision` (kernel-owned, always assigned together in one `hold_sync()`), `selection` (browser-owned, mirrored to `Model.selection`), `height`/`theme`/`read_only`. Requests and replies are custom messages, never traits: browser `{type:'wasm'}` -> bytes as the first binary buffer; browser `{type:'snapshot', id, base, json}` -> `Project._apply_snapshot` -> EXACTLY ONE reply, always, and every reply (`saved`/`rejected`, from whichever arm below, including a malformed snapshot's) echoes the request's `id` verbatim (any JSON value; omitted when the request carried none) -- the browser resolves a save only on the reply naming its request, since one model displayed in several views sees every reply in every view: applied (including a `SimlinWriteError`, where the change is real and a post-commit step failed -- the write, or notifying subscribers) -> traits `(exact json received, base+1)` in one `hold_sync()` that exits BEFORE `{type:'saved'}` is sent (+ a warn `notice` naming what failed; `model.save()` is suggested only when `write_failed`); not applied (stale base, parse failure, or a handler exception BEFORE anything applied) -> no trait writes (the project's real pair is re-pushed, a no-op when unchanged), `{type:'rejected'}` + a warn `notice`; a handler exception AFTER the snapshot applied is an ACCEPT like the write-failure case -- the exact bytes received are pushed at `base+1` (best effort) and `{type:'saved'}` is sent plus a warn `notice`, because a `rejected` there costs the view a remount (undo history and any local edits since are lost) for a change that is real; a failure AFTER the reply went out (`_SnapshotProgress.replied`) sends nothing more, and a `BaseException` (KeyboardInterrupt) gets the same one reply and is then re-raised; browser `{type:'oversize', bytes}` (it refused to send a snapshot whose wire size -- the text JSON-escaped, `_widget_core.snapshot_wire_size` -- exceeds the `max_snapshot_bytes` trait, default `_widget_core.MAX_SNAPSHOT_BYTES` = 8 MiB, chosen to stay under tornado's 10 MiB `websocket_max_message_size` with only the envelope/header left to add) -> a `RuntimeWarning` on the kernel's stderr (a comm handler runs outside any cell; JupyterLab shows it in the Log Console -- the toast is what the user sees) + the same warn `notice` the browser already showed, no reply; seeding or pushing a `project_json` above the cap warns once per widget (`_warn_if_oversize`, attributed to the user's cell by `_widget_core.user_stacklevel()` -- the display, `Model.widget()` call or `edit()` block -- with IPython's display-formatter frames skipped). Subscribes to `Project.on_change` (marshalled to `shell.kernel.io_loop.add_callback` when running under ipykernel, else direct) and pushes the project's current pair plus a notice for every change that is not one of its own accepted snapshots (`_own_revisions`, a SET of the revisions its accepts produced whose notification is still queued -- ipykernel 7 handles comm messages on a subshell thread while a cell runs, so several accepts can precede the first delivery); `close()` unsubscribes. Asset delivery is resolved ONCE at import from `SIMLIN_WIDGET_ASSET` (`bundled` | `inline` | http(s) URL) into `_ASSETS`; a missing asset never breaks `import simlin` -- creating a widget raises `SimlinAssetError`. The `inline` shim contract with the JS (`globalThis.__simlinWidgetInlineWasm = "";` on the line before the module) is documented on `_widget_core.inline_esm` - `simlin/_widget_core.py` -- the widget's pure decisions: `parse_asset_mode`/`inline_esm`/`missing_asset_message`, `parse_incoming` (message classification; malformed messages are `Unrecognised`, never exceptions), `plan_snapshot_reply` (accept / stale / raised arms), the snapshot size guard (`MAX_SNAPSHOT_BYTES` with its rationale, `snapshot_wire_size`, `oversize_warning`/`oversize_report_warning`/`oversize_notice`, `format_size` -- mirrored byte for byte by `snapshotWireSize`/`formatSize` in the JS, with one fixture list pinned in both suites), `is_own_change`, `notice_for_change` (one text per `ChangeSource`), `dispatch_for_shell`. `tests/test_widget_core.py` enumerates every arm; `tests/test_widget.py` drives the shell through a recording comm (no browser, no kernel) -- `e2e/` -- the JupyterLab journey (Playwright; design AC4.2): `global-setup.ts` launches `jupyter lab` from this venv (`jupyter-server.ts`: OS-assigned port, random token, temporary config/data/runtime/root directories, a preflight that the interpreter imports `simlin` with staged widget assets and `jupyterlab`) and hands the URL/token/root to the spec through `SIMLIN_E2E_*` env vars; each notebook's kernel session is started on the server before the browser opens the notebook (`startNotebookSession`, tried `SESSION_START_ATTEMPTS` times), because Lab connects to a session that exists and leaves a notebook kernel-less for good when its own one start request fails; a cell is run only once Lab's own session context says its kernel is connected (`kernelState`, read from `window.jupyterapp`, which the server is launched to expose -- the toolbar's execution indicator reads `idle` before the session has connected to anything, and a run in that window races Lab's session start); each notebook opens in a JupyterLab workspace of its own and is confirmed by path before anything is clicked (`openNotebook`), because Lab restores a shared workspace's earlier notebooks before it opens the one the URL names, and a helper that means "the visible notebook" then runs a cell of the wrong one; `kernel-session.spec.ts` pins the session-start and kernel-state decisions without a browser; `notebook-editor.spec.ts` drives the real JupyterLab UI: `simlin.open` + display -> the Editor renders; a variable added and given an equation in the Editor -> the file on disk changes and a following `m.run()` cell reports it; `m.selection` names the selected variable; a Python `edit()` -> "Updated from Python" in the Editor; a second-process write -> "Updated on disk" and `m.revision` advanced, and after those remounts `m.selection` is `()` again. It is NOT part of `pytest`/pre-commit: `make e2e` runs it (CI: the `pysimlin-e2e` job); it needs the `e2e` extra (jupyterlab, in THIS venv so its share/jupyter carries the anywidget/ipywidgets labextensions), the CFFI extension, staged `simlin/_widget/` assets (`make assets`), and Playwright's chromium. `SIMLIN_E2E_PYTHON` picks another interpreter; `SIMLIN_E2E_KEEP_TMP` keeps the server tree; the server log (jupyter-lab.log) is written to the git-ignored output directory beside the spec (see `e2e/playwright.config.ts`) +- `e2e/` -- the JupyterLab journey (Playwright; design AC4.2): `global-setup.ts` launches `jupyter lab` from this venv (`jupyter-server.ts`: OS-assigned port, random token, temporary config/data/runtime/root directories, a preflight that the interpreter imports `simlin` with staged widget assets and `jupyterlab`, and the runtime file read only once whole -- `readServerInfo`, since the server writes it in place) and hands the URL/token/root/server-log path to the spec through `SIMLIN_E2E_*` env vars; each notebook's kernel session is started on the server before the browser opens the notebook (`startNotebookSession`, tried `SESSION_START_ATTEMPTS` times), because Lab connects to a session that exists and leaves a notebook kernel-less for good when its own one start request fails; a cell is run only once Lab's own session context says its kernel is connected (`kernelState`, read from `window.jupyterapp`, which the server is launched to expose -- the toolbar's execution indicator reads `idle` before the session has connected to anything, and a run in that window races Lab's session start); a cell run is an action with a checked effect: `runCell` waits for the prompt to show a new execution count (`parsePrompt`/`cellRan`) and otherwise fails at the run with `unrunCellReport` (`[*]` = sent and unanswered by the kernel, no new count = Lab never ran it; the kernel's state and the server log's tail) rather than as an output that never appears; the `e2e` extra requires ipykernel >= 7.4, the first release that sends shell replies through the shell socket's ZMQStream (ipython/ipykernel#1529) -- before it a reply sent raw on that socket could strand a request arriving at the same moment, Lab's burst of requests when it connects a notebook made that likely on a notebook's first cell, and the kernel then sat idle with the cell at `[*]` for good; each notebook opens in a JupyterLab workspace of its own and is confirmed by path before anything is clicked (`openNotebook`), because Lab restores a shared workspace's earlier notebooks before it opens the one the URL names, and a helper that means "the visible notebook" then runs a cell of the wrong one; `kernel-session.spec.ts` pins the runtime-file, session-start, kernel-state, and cell-run decisions without a browser; `notebook-editor.spec.ts` drives the real JupyterLab UI: `simlin.open` + display -> the Editor renders; a variable added and given an equation in the Editor -> the file on disk changes and a following `m.run()` cell reports it; `m.selection` names the selected variable; a Python `edit()` -> "Updated from Python" in the Editor; a second-process write -> "Updated on disk" and `m.revision` advanced, and after those remounts `m.selection` is `()` again. It is NOT part of `pytest`/pre-commit: `make e2e` runs it (CI: the `pysimlin-e2e` job); it needs the `e2e` extra (jupyterlab, in THIS venv so its share/jupyter carries the anywidget/ipywidgets labextensions), the CFFI extension, staged `simlin/_widget/` assets (`make assets`), and Playwright's chromium. `SIMLIN_E2E_PYTHON` picks another interpreter; `SIMLIN_E2E_KEEP_TMP` keeps the server tree; the server log (jupyter-lab.log) is written to the git-ignored output directory beside the spec (see `e2e/playwright.config.ts`) - `scripts/check_notebook_export.py` -- executes an example notebook headless with nbconvert and checks both static-export arms of the display cell: with the stored widget state the HTML carries the widget view plus the embedded state (the `_esm` rides along), and with `metadata.widgets` stripped -- what GitHub or a notebook saved without state gives a static renderer -- nbconvert's `WidgetsDataTypeFilter` skips the widget-view mimetype and the HTML carries the SVG (the lab template embeds it as a base64 ``). Its pure half is unit-tested in `tests/test_check_notebook_export.py`; the shell runs under `make export-check` and the CI `pysimlin-e2e` job, never as a pytest (the dev extra has no nbconvert and a skipped test proves nothing) - `docs/notebook-hosts.md` -- per-host checklist for the notebook editor (JupyterLab 4, Notebook 7, VS Code local/Remote-SSH, Colab, marimo) with each host's VERIFIED/UNVERIFIED status; only JupyterLab is verified (by `e2e/`). Update the status line in the same PR that runs a checklist. `examples/colab_quickstart.ipynb` is the Colab entry point (its `%pip` cell is skipped by `tests/test_examples.py`'s script run) - `simlin/_widget/` -- package data: `widget.js` and `libsimlin-browser.wasm`, build outputs of the `@simlin/notebook-widget` TypeScript package (git-ignored; see its README.md and `pyproject.toml` package-data) diff --git a/src/pysimlin/e2e/global-setup.ts b/src/pysimlin/e2e/global-setup.ts index fe1a10620..ea3d15421 100644 --- a/src/pysimlin/e2e/global-setup.ts +++ b/src/pysimlin/e2e/global-setup.ts @@ -19,4 +19,5 @@ export default async function globalSetup(): Promise { process.env[ENV.url] = server.url; process.env[ENV.token] = server.token; process.env[ENV.rootDir] = server.rootDir; + process.env[ENV.logPath] = server.logPath; } diff --git a/src/pysimlin/e2e/jupyter-server.ts b/src/pysimlin/e2e/jupyter-server.ts index ba09aa5be..198bdb2ad 100644 --- a/src/pysimlin/e2e/jupyter-server.ts +++ b/src/pysimlin/e2e/jupyter-server.ts @@ -38,6 +38,7 @@ export const ENV = { url: 'SIMLIN_E2E_JUPYTER_URL', token: 'SIMLIN_E2E_JUPYTER_TOKEN', rootDir: 'SIMLIN_E2E_ROOT_DIR', + logPath: 'SIMLIN_E2E_SERVER_LOG', } as const; const here = __dirname; @@ -93,7 +94,28 @@ export function userSettingsFiles(configDir: string): Array<{ file: string; cont ]; } -/** Read `url`/`token`/`root_dir` from a `jpserver-.json` runtime file. */ +/** + * Read `url`/`token`/`root_dir` from a `jpserver-.json` runtime file, + * or `undefined` while the file is still being written. + * + * The server writes the file in place (jupyter_server's `secure_write` opens + * the final path and `json.dump`s into it), so a reader that finds the file + * can read it truncated, or empty. Text that does not parse yet is that + * window; a file that parses but lacks a field is a real error. + */ +export function readServerInfo(json: string): ServerInfo | undefined { + try { + JSON.parse(json); + } catch (err) { + if (err instanceof SyntaxError) { + return undefined; + } + throw err; + } + return parseServerInfo(json); +} + +/** Read `url`/`token`/`root_dir` from a complete `jpserver-.json` runtime file. */ export function parseServerInfo(json: string): ServerInfo { const info = JSON.parse(json) as { url?: unknown; token?: unknown; root_dir?: unknown }; if (typeof info.url !== 'string' || typeof info.token !== 'string' || typeof info.root_dir !== 'string') { @@ -266,6 +288,77 @@ export function canRunCells(state: string): boolean { return state === 'idle' || state === 'busy'; } +/** + * A code cell's execution as its input prompt shows it: `null` for never run + * (`[ ]:`), `'running'` for sent to the kernel and not yet answered (`[*]:`), + * or the execution count the kernel replied with (`[3]:`). + */ +export type CellExecution = null | 'running' | number; + +/** Read a cell's input prompt text (`[ ]:`, `[*]:`, `[3]:`). */ +export function parsePrompt(text: string): CellExecution { + const m = /^\s*\[\s*([^\]]*?)\s*\]:?\s*$/.exec(text); + if (m === null) { + throw new Error(`not a cell input prompt: ${JSON.stringify(text)}`); + } + if (m[1] === '') { + return null; + } + if (m[1] === '*') { + return 'running'; + } + if (/^\d+$/.test(m[1])) { + return Number(m[1]); + } + throw new Error(`not a cell input prompt: ${JSON.stringify(text)}`); +} + +/** + * Whether a run of a cell whose prompt read `before` has been answered by the + * kernel, now that it reads `now`: the kernel's reply carries a new execution + * count, so a count that differs from the one the cell had is the kernel's + * word that it ran the cell. A cell that shows no count, or `[*]`, or the + * count it had before, has not been run by this press. + */ +export function cellRan(before: CellExecution, now: CellExecution): boolean { + return typeof now === 'number' && now !== before; +} + +/** + * Why a cell run the journey asked for never came back, in terms that say + * which side to look at: a `[*]` prompt means JupyterLab sent the request and + * the kernel has not answered (look at the kernel and the server log), and a + * prompt with no new count means JupyterLab never ran the cell at all (look at + * the page: focus, a dialog, a cancelled run). + */ +export function unrunCellReport(facts: { + notebook: string; + index: number; + before: CellExecution; + now: CellExecution; + kernel: string; + waitedMs: number; + serverLogTail: string; +}): string { + const shown = (e: CellExecution): string => (e === null ? '[ ]' : e === 'running' ? '[*]' : `[${e}]`); + const verdict = + facts.now === 'running' + ? 'JupyterLab sent it to the kernel, which has not answered' + : 'JupyterLab never ran it (no execution request went out, or the run was cancelled)'; + return [ + `cell ${facts.index} of ${facts.notebook} did not run within ${facts.waitedMs} ms: ${verdict}.`, + `prompt before the run: ${shown(facts.before)}; now: ${shown(facts.now)}; notebook kernel: ${facts.kernel}`, + `last lines of the jupyter server log:`, + facts.serverLogTail, + ].join('\n'); +} + +/** The last `lines` lines of `text`, for an error message. */ +export function tail(text: string, lines: number): string { + const all = text.trimEnd().split('\n'); + return all.slice(Math.max(0, all.length - lines)).join('\n'); +} + /** Launch JupyterLab and resolve once its REST API answers with our token. */ export async function launchJupyterLab(): Promise { const python = pythonExecutable(); @@ -322,7 +415,7 @@ export async function launchJupyterLab(): Promise { if (files.length === 0) { return undefined; } - return parseServerInfo(fs.readFileSync(path.join(runtimeDir, files[0]), 'utf8')); + return readServerInfo(fs.readFileSync(path.join(runtimeDir, files[0]), 'utf8')); }); const url = info.url; const bearer = info.token; diff --git a/src/pysimlin/e2e/kernel-session.spec.ts b/src/pysimlin/e2e/kernel-session.spec.ts index b087c5462..3d8055419 100644 --- a/src/pysimlin/e2e/kernel-session.spec.ts +++ b/src/pysimlin/e2e/kernel-session.spec.ts @@ -3,12 +3,16 @@ // Version 2.0, that can be found in the LICENSE file. /** - * The two decisions the journey makes about a notebook's kernel, apart from - * any browser: how its session is started (`startNotebookSession`: one - * request per attempt, a refusal tried again, a server that refuses every - * time reported with each refusal's own words) and when a cell run would - * execute (`kernelState`, `canRunCells`). The server here is a stand-in for - * `fetch` and the session facts are written out; the journey itself exercises + * The decisions the journey makes about its server and a notebook's kernel, + * apart from any browser: when the server's runtime file can be read + * (`readServerInfo`: a file still being written is waited out, a complete one + * missing a field is an error); how a notebook's session is started + * (`startNotebookSession`: one request per attempt, a refusal tried again, a + * server that refuses every time reported with each refusal's own words); + * when a cell run would execute (`kernelState`, `canRunCells`); and whether a + * run was answered, and what to say when it was not (`parsePrompt`, + * `cellRan`, `unrunCellReport`). The server here is a stand-in for `fetch` and + * the session facts and prompts are written out; the journey itself exercises * the real ones. */ @@ -18,9 +22,15 @@ import { KERNEL_NAME, SESSION_START_ATTEMPTS, canRunCells, + cellRan, kernelState, + parsePrompt, + readServerInfo, sessionRequestBody, startNotebookSession, + tail, + unrunCellReport, + type CellExecution, type SessionFacts, } from './jupyter-server'; @@ -118,3 +128,76 @@ test('a cell run executes only in a started session with a connected, live kerne expect(canRunCells(state), state).toBe(runs); } }); + +const SERVER_INFO = JSON.stringify({ url: 'http://127.0.0.1:8888/', token: 't', root_dir: '/r', pid: 1 }, null, 2); + +test('the runtime file is read once it is whole, and waited for while it is being written', () => { + expect(readServerInfo(SERVER_INFO)).toEqual({ url: 'http://127.0.0.1:8888/', token: 't', rootDir: '/r' }); + // The server writes the file in place: an empty file, and every truncation + // of a whole one, is the write still under way. + for (let n = 0; n < SERVER_INFO.length; n++) { + expect(readServerInfo(SERVER_INFO.slice(0, n)), `first ${n} characters`).toBeUndefined(); + } + // Whole but without a field the journey needs: no amount of waiting fixes it. + expect(() => readServerInfo(JSON.stringify({ url: 'http://127.0.0.1:8888/', token: 't' }))).toThrow( + /unexpected jupyter server info/, + ); +}); + +// Every prompt a code cell shows, as JupyterLab renders it (the empty prompt +// carries a non-breaking space). +const PROMPTS: Array<{ text: string; execution: CellExecution }> = [ + { text: '[\u00a0]:', execution: null }, + { text: '[ ]:', execution: null }, + { text: '[*]:', execution: 'running' }, + { text: '[1]:', execution: 1 }, + { text: '[42]:', execution: 42 }, +]; + +test('a cell prompt reads as never run, running, or the execution count', () => { + for (const { text, execution } of PROMPTS) { + expect(parsePrompt(text), JSON.stringify(text)).toBe(execution); + } + for (const text of ['', 'In [1]:', '[x]:', '[1']) { + expect(() => parsePrompt(text), JSON.stringify(text)).toThrow(/not a cell input prompt/); + } +}); + +// Every (before, now) pair of execution states: a run is answered only when +// the prompt shows a count it did not show before. +const EXECUTIONS: CellExecution[] = [null, 'running', 3, 4]; + +test('a run is answered exactly when the prompt shows a new execution count', () => { + for (const before of EXECUTIONS) { + for (const now of EXECUTIONS) { + const answered = typeof now === 'number' && now !== before; + expect(cellRan(before, now), `${String(before)} -> ${String(now)}`).toBe(answered); + } + } +}); + +test('an unanswered run says which side never acted, with the prompt, the kernel, and the server log', () => { + const facts = { + notebook: 'scratch.ipynb', + index: 0, + before: null, + kernel: 'idle', + waitedMs: 60_000, + serverLogTail: '[I ServerApp] Connecting to kernel k.', + }; + const sent = unrunCellReport({ ...facts, now: 'running' }); + expect(sent).toContain('cell 0 of scratch.ipynb did not run within 60000 ms'); + expect(sent).toContain('JupyterLab sent it to the kernel, which has not answered'); + expect(sent).toContain('prompt before the run: [ ]; now: [*]; notebook kernel: idle'); + expect(sent).toContain('[I ServerApp] Connecting to kernel k.'); + for (const now of [null, 3] as CellExecution[]) { + const unsent = unrunCellReport({ ...facts, before: 3, now }); + expect(unsent).toContain('JupyterLab never ran it'); + expect(unsent).toContain(`prompt before the run: [3]; now: ${now === null ? '[ ]' : '[3]'}`); + } +}); + +test('a log tail keeps the last lines', () => { + expect(tail('a\nb\nc\n', 2)).toBe('b\nc'); + expect(tail('a\nb', 5)).toBe('a\nb'); +}); diff --git a/src/pysimlin/e2e/notebook-editor.spec.ts b/src/pysimlin/e2e/notebook-editor.spec.ts index 6edc7d1df..12284ac8f 100644 --- a/src/pysimlin/e2e/notebook-editor.spec.ts +++ b/src/pysimlin/e2e/notebook-editor.spec.ts @@ -33,10 +33,14 @@ import { test, expect, type Locator, type Page } from '@playwright/test'; import { ENV, canRunCells, + cellRan, kernelState, + parsePrompt, pysimlinDir, pythonExecutable, startNotebookSession, + tail, + unrunCellReport, type SessionFacts, } from './jupyter-server'; @@ -252,12 +256,56 @@ async function waitForKernel(page: Page): Promise { } } -/** Click into cell `index`'s editor and run it with Shift+Enter. */ +/** How long a cell run may take to be answered; the journey's cells take seconds. */ +const CELL_RUN_TIMEOUT_MS = 60_000; + +/** + * Click into cell `index`'s editor, run it with Shift+Enter, and wait until + * the kernel has answered the run (`cellRan`: the prompt shows a new + * execution count). + * + * A run that is never answered otherwise surfaces later as an output that + * never appears, which says nothing about why; this fails at the run instead, + * with what the page and the server know (`unrunCellReport`). + */ async function runCell(page: Page, index: number): Promise { await waitForKernel(page); const cell = activeCells(page).nth(index); + const prompt = cell.locator('.jp-InputPrompt'); + const before = parsePrompt((await prompt.textContent()) ?? ''); await cell.locator('.jp-InputArea-editor').click(); await page.keyboard.press('Shift+Enter'); + let now = before; + try { + await expect + .poll( + async () => { + now = parsePrompt((await prompt.textContent()) ?? ''); + return cellRan(before, now); + }, + { timeout: CELL_RUN_TIMEOUT_MS }, + ) + .toBe(true); + } catch { + const notebook = (await visibleNotebookPath(page)) ?? 'the visible notebook'; + let serverLog: string; + try { + serverLog = tail(fs.readFileSync(required(ENV.logPath), 'utf8'), 20); + } catch (err) { + serverLog = `(unreadable: ${err instanceof Error ? err.message : String(err)})`; + } + throw new Error( + unrunCellReport({ + notebook, + index, + before, + now, + kernel: kernelState(await sessionFacts(page)), + waitedMs: CELL_RUN_TIMEOUT_MS, + serverLogTail: serverLog, + }), + ); + } return cell; } diff --git a/src/pysimlin/pyproject.toml b/src/pysimlin/pyproject.toml index 173595bb8..2e6fcf965 100644 --- a/src/pysimlin/pyproject.toml +++ b/src/pysimlin/pyproject.toml @@ -96,9 +96,18 @@ notebooks = [ # never import. It must live in THIS venv (not `uv run --with jupyterlab`, # whose ephemeral prefix hides the anywidget/ipywidgets labextensions that # ship under this venv's share/jupyter). +# +# ipykernel 7.4 is the first release that sends shell replies through the +# shell socket's ZMQStream (ipython/ipykernel#1529). Before it, a reply sent +# raw on that socket could consume the edge of a request arriving at the same +# moment, and the kernel then sat idle with the request unread, for good. +# Lab's burst of requests when it connects a notebook (history, debug, the +# widget manager's comm) makes that race likely on a notebook's first cell: +# the cell shows [*] and never runs. e2e = [ "pysimlin[notebook]", "jupyterlab>=4.3", + "ipykernel>=7.4", ] [project.urls] diff --git a/src/pysimlin/uv.lock b/src/pysimlin/uv.lock index 5e6aa5db9..7ead11f25 100644 --- a/src/pysimlin/uv.lock +++ b/src/pysimlin/uv.lock @@ -816,7 +816,7 @@ wheels = [ [[package]] name = "ipykernel" -version = "7.2.0" +version = "7.4.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "appnope", marker = "sys_platform == 'darwin'" }, @@ -826,16 +826,15 @@ dependencies = [ { name = "jupyter-client" }, { name = "jupyter-core" }, { name = "matplotlib-inline" }, - { name = "nest-asyncio" }, + { name = "nest-asyncio2" }, { name = "packaging" }, - { name = "psutil" }, { name = "pyzmq" }, { name = "tornado" }, { name = "traitlets" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ca/8d/b68b728e2d06b9e0051019640a40a9eb7a88fcd82c2e1b5ce70bef5ff044/ipykernel-7.2.0.tar.gz", hash = "sha256:18ed160b6dee2cbb16e5f3575858bc19d8f1fe6046a9a680c708494ce31d909e", size = 176046, upload-time = "2026-02-06T16:43:27.403Z" } +sdist = { url = "https://files.pythonhosted.org/packages/7a/05/e5499c8b762387d83926bf0389e8fff888c9bf6e7af30566a8527bc12f4a/ipykernel-7.4.0.tar.gz", hash = "sha256:4330114d22b9b33575b2c7c68753fc8faabbbcfbc60b0cdff9f14dd4ce63742f", size = 192731, upload-time = "2026-09-29T16:33:11.468Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/82/b9/e73d5d9f405cba7706c539aa8b311b49d4c2f3d698d9c12f815231169c71/ipykernel-7.2.0-py3-none-any.whl", hash = "sha256:3bbd4420d2b3cc105cbdf3756bfc04500b1e52f090a90716851f3916c62e1661", size = 118788, upload-time = "2026-02-06T16:43:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/a5/f8/8b5c10c7f36ab5a3b5c4703669e5b14e44e98782b58e535dd30670e8799a/ipykernel-7.4.0-py3-none-any.whl", hash = "sha256:a6757f790ddc5a6006b813d82da1a9dbb65b3a00f2b6607df776418c6566ec1c", size = 123184, upload-time = "2026-09-29T16:33:09.612Z" }, ] [[package]] @@ -998,7 +997,7 @@ wheels = [ [[package]] name = "jupyter-client" -version = "8.8.0" +version = "8.10.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jupyter-core" }, @@ -1006,10 +1005,11 @@ dependencies = [ { name = "pyzmq" }, { name = "tornado" }, { name = "traitlets" }, + { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/05/e4/ba649102a3bc3fbca54e7239fb924fd434c766f855693d86de0b1f2bec81/jupyter_client-8.8.0.tar.gz", hash = "sha256:d556811419a4f2d96c869af34e854e3f059b7cc2d6d01a9cd9c85c267691be3e", size = 348020, upload-time = "2026-01-08T13:55:47.938Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c5/2a/906772148a06e48885039e0250c340b770a55bbf37b08d6ee0449df369c3/jupyter_client-8.10.0.tar.gz", hash = "sha256:9f7116294dca55f1785be880057d44544db9b1567718d92cb33c58886afb9497", size = 360653, upload-time = "2026-08-28T12:17:10.854Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2d/0b/ceb7694d864abc0a047649aec263878acb9f792e1fec3e676f22dc9015e3/jupyter_client-8.8.0-py3-none-any.whl", hash = "sha256:f93a5b99c5e23a507b773d3a1136bd6e16c67883ccdbd9a829b0bbdb98cd7d7a", size = 107371, upload-time = "2026-01-08T13:55:45.562Z" }, + { url = "https://files.pythonhosted.org/packages/0f/88/7c548de1f6c2ade7c931a3282da73f9274fa6a1531091be682f89c85efb9/jupyter_client-8.10.0-py3-none-any.whl", hash = "sha256:5f73f24f22fa25192cfff6b23c051932a2473a797b05734aff495b392103e14e", size = 110184, upload-time = "2026-08-28T12:17:09.028Z" }, ] [[package]] @@ -1601,12 +1601,12 @@ wheels = [ ] [[package]] -name = "nest-asyncio" -version = "1.6.0" +name = "nest-asyncio2" +version = "1.7.3" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/83/f8/51569ac65d696c8ecbee95938f89d4abf00f47d58d48f6fbabfe8f0baefe/nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe", size = 7418, upload-time = "2024-01-21T14:25:19.227Z" } +sdist = { url = "https://files.pythonhosted.org/packages/5e/a6/a2775b388a14b5ea0c894bbe513c0d3e04e65e8ae583e60f5357722b8e03/nest_asyncio2-1.7.3.tar.gz", hash = "sha256:2e9a84d5d1efe6d020c72988d21aec569bac42d98af2ff6b9de24640c5d22a34", size = 15642, upload-time = "2026-09-22T17:15:06.709Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a0/c4/c2971a3ba4c6103a3d10c4b0f24f461ddc027f0f09763220cf35ca1401b3/nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c", size = 5195, upload-time = "2024-01-21T14:25:17.223Z" }, + { url = "https://files.pythonhosted.org/packages/29/dd/a8bebc78975a7466932077a0973cbab4361a32368b1d28d40683ec19333c/nest_asyncio2-1.7.3-py3-none-any.whl", hash = "sha256:2bc87bdca654e719425145f5e84eaeb0080b013fdd47d65729a7d66243f4987c", size = 8364, upload-time = "2026-09-22T17:15:05.492Z" }, ] [[package]] @@ -2089,6 +2089,7 @@ dev = [ ] e2e = [ { name = "anywidget" }, + { name = "ipykernel" }, { name = "jupyterlab" }, ] notebook = [ @@ -2117,6 +2118,7 @@ requires-dist = [ { name = "cattrs", specifier = ">=23" }, { name = "cffi", specifier = ">=1.15.0" }, { name = "hypothesis", marker = "extra == 'test'", specifier = ">=6.100.0" }, + { name = "ipykernel", marker = "extra == 'e2e'", specifier = ">=7.4" }, { name = "ipykernel", marker = "extra == 'notebooks'", specifier = ">=6.29" }, { name = "jsonschema", marker = "extra == 'test'", specifier = ">=4.20.0" }, { name = "jupyterlab", marker = "extra == 'e2e'", specifier = ">=4.3" }, @@ -2685,19 +2687,19 @@ wheels = [ [[package]] name = "tornado" -version = "6.5.6" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/50/57/6d7303a77ae439d9189108f76c0c4fd89ee5e2cc8387bffb55232565c4ed/tornado-6.5.6.tar.gz", hash = "sha256:9a365179fe8ff6b8766f602c0f67c185d778193e9bdd828b19f0b6ed7764177d", size = 518139, upload-time = "2026-05-27T15:35:54.646Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/1b/0d/b4f481e18c5a51864e6d12b9a05ecf72919696680b747c958c3fc1f4fbae/tornado-6.5.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:65fcfaafb079435c2c19dc9e07c0f1cf0fa9051759ed0a7d0a3ba7ea7f64919c", size = 447737, upload-time = "2026-05-27T15:35:38.122Z" }, - { url = "https://files.pythonhosted.org/packages/9e/9c/5430c39fcab1144d35860f457b15e9c08b4bc7ac86764354204e983d6183/tornado-6.5.6-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:38bc01b4acacded2de63ae78023548e41ebe6fbed3ec05a796d7ae3ad893887e", size = 445899, upload-time = "2026-05-27T15:35:40.519Z" }, - { url = "https://files.pythonhosted.org/packages/8b/79/fa7e14a2f939c807a8d30619b4eb604eab219601b78792516ebe22d40cf9/tornado-6.5.6-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b942e6a137fda31ff54bf8e6e2c8d1c37f1f50583f3ed53fb840b53b9601d104", size = 448964, upload-time = "2026-05-27T15:35:42.106Z" }, - { url = "https://files.pythonhosted.org/packages/a7/71/bd67d5f5199f937dafe03a49a37989f60f600ff6fef34c79412a829d97bd/tornado-6.5.6-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8666946e70171b8c3f1fc9b7876fac492e84822c4c7f3746f4e8f8bc9ac92a79", size = 449935, upload-time = "2026-05-27T15:35:43.906Z" }, - { url = "https://files.pythonhosted.org/packages/cc/a4/c24388c9cf5b3c3a513b56a158af9f23092c9a2810d789e294310797df21/tornado-6.5.6-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1c34cfab7ad6d104f052f55de06d39bbafc5885cfeb4da688803308dbcfa90b7", size = 449767, upload-time = "2026-05-27T15:35:45.793Z" }, - { url = "https://files.pythonhosted.org/packages/a5/eb/6a07ad550c3f7b37244bd0becdf293ec3d3e961783d8b720a97df50de1b2/tornado-6.5.6-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:385f35e4e22fb52551dfcda4cdc8c30c61c2c001aef5ddad99cdfe116952efd3", size = 449174, upload-time = "2026-05-27T15:35:47.485Z" }, - { url = "https://files.pythonhosted.org/packages/bb/84/3469e098dccdb6763130e06aacd786bb4363fca7b590a55c101ddf34ed30/tornado-6.5.6-cp39-abi3-win32.whl", hash = "sha256:db475f1b67b2809b10bb16264829087724ca8d24fe4ed47f7b8675cae453ef86", size = 450230, upload-time = "2026-05-27T15:35:49.322Z" }, - { url = "https://files.pythonhosted.org/packages/d2/3c/273a04e0b9dd9016f1685cca0c1c8795a71ac88a34a8c889a0b443483226/tornado-6.5.6-cp39-abi3-win_amd64.whl", hash = "sha256:6739bf1e8eb09230f1280ddbd3236f0309db70f2c551a8dbc40f62babdf82f79", size = 450667, upload-time = "2026-05-27T15:35:51.194Z" }, - { url = "https://files.pythonhosted.org/packages/02/98/0cffe22a224f60c5fb1e3aa0b76f9da2e1ca78b0e9545e3d077c68ce60a7/tornado-6.5.6-cp39-abi3-win_arm64.whl", hash = "sha256:2543597b24a695d72338a9a77818362d72387c03ae173f1f169eadc5c91466ac", size = 449690, upload-time = "2026-05-27T15:35:52.902Z" }, +version = "6.5.10" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/06/61/53d562a57b28c08eda40b258c0f975e360541943ad7c7bef897a40caafda/tornado-6.5.10.tar.gz", hash = "sha256:a6b1ccd08c04b4a06fb5aeb381be99de5ad1e5375c1785e31d78c880feb57687", size = 537910, upload-time = "2026-09-15T13:47:48.73Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cd/5b/ff5fc58fa2427c30dea74c90053f4fc5eda1e7f3833ed3ecc7147fe2b311/tornado-6.5.10-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9261783640e23258694a9ff0795df430a5a7b0a651d3dd53dd0969ad6be16da7", size = 465883, upload-time = "2026-09-15T13:47:35.463Z" }, + { url = "https://files.pythonhosted.org/packages/ad/f5/cd7be26c34a3315532f3aef5f092465da8f59c334dd439d3c14aaef16461/tornado-6.5.10-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:83e6cf438b106c6b3852d70960967bb1b70c87438050dca0981e4b9aa751a4c1", size = 464046, upload-time = "2026-09-15T13:47:37.178Z" }, + { url = "https://files.pythonhosted.org/packages/60/33/df6d7d04854a58619f8349a51e3edb138324130a7562b0bb21f115bb940f/tornado-6.5.10-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bdf942448169e5336451d0494d7e3d81cfa726d5aa312affdc4682dd62a62f6d", size = 467096, upload-time = "2026-09-15T13:47:38.559Z" }, + { url = "https://files.pythonhosted.org/packages/29/17/cc35dff68272d685cffd8600ffafbd8067e7d05e7348d9f80caddffbbd5f/tornado-6.5.10-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:69acca6501eed74582b76dbbceee2a91613f54728e3e418346000d7103101676", size = 468067, upload-time = "2026-09-15T13:47:40.085Z" }, + { url = "https://files.pythonhosted.org/packages/c3/01/6e5349b4e1a53a4b4972a6716785e1fe7407f312063c3972690af8ff301b/tornado-6.5.10-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:66aaa3f57d30c6e6becee83ff28055d5930ac724214bde99393eefda83d5e015", size = 467901, upload-time = "2026-09-15T13:47:41.576Z" }, + { url = "https://files.pythonhosted.org/packages/28/5e/b4facf94370dba006819c8d304376f8b9fbec6b935b5e51bf45823a9790b/tornado-6.5.10-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4bd192b959f9128fb99b8898148070ba4574c9589b78bce42d1851131fe85828", size = 467308, upload-time = "2026-09-15T13:47:43.145Z" }, + { url = "https://files.pythonhosted.org/packages/56/ae/047938e828cafc8eca4c908fafb6588fee944e3af39a0af9d7b602499ae5/tornado-6.5.10-cp39-abi3-win32.whl", hash = "sha256:302eb1e0e3e159314eb591920529fdea80acca92df5510a2cec5bbd4f099ec72", size = 468387, upload-time = "2026-09-15T13:47:44.556Z" }, + { url = "https://files.pythonhosted.org/packages/d8/d4/5901517f05affd752490f6a654ba31b7474664e8dd80bd045a00c220bd88/tornado-6.5.10-cp39-abi3-win_amd64.whl", hash = "sha256:37ae8f150cecfdbf747fc4e12f5e9a97ecd8cf1d4cdb3f119e2de84b11196918", size = 468828, upload-time = "2026-09-15T13:47:45.961Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1a/fd497f3a7f7b74bb04f4b94536b5c9f80742b5d50501fd27977652ddec16/tornado-6.5.10-cp39-abi3-win_arm64.whl", hash = "sha256:ce045d3c298fddd30e89a2777f97039d1b641eb9518ac7b26a4721903539c694", size = 467847, upload-time = "2026-09-15T13:47:47.283Z" }, ] [[package]]