Skip to content

Reduce first-evaluation latency for R and Python sessions - #188

Draft
t-kalinowski wants to merge 19 commits into
mainfrom
perf/first-evaluation
Draft

t-kalinowski wants to merge 19 commits into
mainfrom
perf/first-evaluation

Conversation

@t-kalinowski

@t-kalinowski t-kalinowski commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

First evaluation currently pays for Python initialization and the default SQL connection even when neither is used. This change initializes them at their first consumer while retaining eager configured startup programs and native R startup.

r.enabled: false skips R discovery, preparation, and initialization, including on hosts with R installed. Python-only and Python/SQL sessions support managed or existing environments; an existing environment with R disabled no longer needs a retained preparation process. The documented Python/SQL configuration uses r.enabled: false, python: .venv, and languages: [python, sql].

Internally, disabled R package preparation skips an unused reticulate bootstrap probe. Managed Python reports its interpreter and prepares its optional Matplotlib cache in one child process. Matplotlib and DuckDB preparation reuse successful receipts only while the runtime, packages, preparation source, and cache identities still match. Small R preparation children omit default packages. Interpreter selection, startup-hook ordering, callback ownership, cancellation, and worker retirement have public regression coverage. Setup and restart regressions record progress only after their interpreter setup checkpoints, preserving exact output and retirement assertions. Transcript collectors also retain the final [done] marker when control-and-cell output spans replies; a gated native R startup regression and related restart continuations verify this completion contract.

On Windows, Console updates the shared C runtime’s environment view when Python initializes and when R startup hints are restored. This preserves Unicode startup-hook paths and keeps the managed environment selection visible to R after native startup. Native startup and interrupt fixtures select Python-first or R-first initialization explicitly. Portable fixtures compare interpreter-directory identity and share exact discovery transcripts for their execution mode.

On Windows, a Python startup hook that starts R cannot initialize reticulate from an R profile before Console’s adapter exists. Reticulate does not detect the running Python interpreter in this path. A native Windows acceptance case preserves the complete diagnostic; the macOS/Linux case continues to assert successful nested startup. The native and transcript acceptance cases share a dependency-light public startup fixture, so native acceptance does not import the transcript renderer or its YAML dependencies.

The benchmark report includes the harness, all timed samples, runtime versions, and binary hashes. On the measured macOS host with warm dependency caches, direct R startup with an existing Python environment and no uv fell from 1,257 to 373 ms; managed Python-only startup fell from 635 to 477 ms. Existing Python without DuckDB showed no consistent improvement. The fastest measured R configuration still takes about 1.44 times plain R, so the 10–20% overhead target remains unmet.

Local macOS validation covers all nine affected public cases (15 executions), the final shared-deadline startup case, and all 23 MCP client self-tests on identical client and collector sources. Formatting and scripts/check passed; the final fixture-only adjustments were validated by their focused public cases.

Broader local macOS coverage includes the runtime-bootstrap and progress suites (22 cases, 41 executions), 60 focused runtime cases (107 executions), all 36 website checks, and both POSIX nested-startup cases in direct and Seatbelt modes. The public nested-startup scenario also passed with Python site packages disabled. The shared-CRT synchronization function typechecks against the repository dependencies for x86_64-pc-windows-msvc; native Windows is unavailable locally.

Hosted checks for the current test-only repair are pending.

Combine interpreter discovery and Matplotlib warming in uv's selected child.
Reuse successful font and DuckDB preparation when current runtime, package,
recipe, and cache identities match, and omit default R packages from the
final preparation children.

Cover reuse, invalidation, failure, and cancellation through public sessions.
Keep configured startup and native R startup eager, while deferring unused
Python inspection, Python setup, and default SQL connections. Skip unused
R preparation probes and bypass R discovery entirely with r.enabled=false.

Preserve the captured interpreter across native startup, retain site-hook
ordering and reentrant R callbacks, and keep cancellation and retirement
ownership intact. Cover Python-only, Python with SQL, and existing environments
through public sessions.
Compare fresh public MCP sessions with plain interpreter startup across
existing and managed environments, with R enabled, absent, or disabled.
Record completed-send latency, warm calls, cache policy, runtime metadata,
and binary provenance with interleaved baseline and candidate samples.
Apply the existing test-only DuckDB snapshot to direct and MXC SQL
scenarios. Assert the selected version before the long query so cold
caches cannot silently introduce the known Arrow cancellation limitation.
Preserve production defaults, interruption deadlines, and exact results.
Queue stdin while the replacement cell is held at its checkpoint, then
release it and collect the complete result. Preserve the exact output
and idle-state assertions without assuming one response completes the cell.
Keep the required completion marker when a control-and-cell response is
split across polls. Exercise the split with a gated native R startup cell
and select the same contract in related restart continuations.
Select the controlled completion contract for the replacement cell so
separate output and completion replies preserve the final marker.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant