Skip to content

feat: add render_pygwalker() for PyGWalker data exploration - #289

Merged
cpsievert merged 1 commit into
render-embedding-atlasfrom
render-pygwalker
Oct 27, 2026
Merged

cpsievert merged 1 commit into
render-embedding-atlasfrom
render-pygwalker

Conversation

@cpsievert

@cpsievert cpsievert commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

@render_pygwalker renders PyGWalker's Tableau-style drag-and-drop explorer as a Shiny output. Return a pandas DataFrame (a polars one works when the installed pygwalker can read it — see below), or a widget from pygwalker.api.anywidget.walk() to pass PyGWalker options like theme_key. The explorer computes in the Python process, fills its card or container (scrolling instead of clipping when the container is shorter than its ~750px UI), and re-renders reactively — see examples/pygwalker for a dataset switcher.

Part of #278.

Security model

PyGWalker runs the SQL its browser sends on DuckDB's process-wide default connection, shared by every PyGWalker widget in the process under one fixed view name. A Shiny app can't safely expose that as-is, so this renderer:

  • Requires the connection to be locked down before rendering (enable_external_access = false + lock_configuration = true), raising with instructions otherwise. This blocks SQL that reaches the server's filesystem or network.
  • Validates visitor SQL (get_datas, batch_get_datas_by_sql, export_dataframe_by_sql) with sqlglot: only single SELECT statements against the widget's own data. Without this, any visitor could persist DDL in the shared in-memory database for every other visitor to read, or query anything else registered on the connection.
  • Rejects the actions that can't be made safe: open_in_desktop (shells out to the server's gw:// handler) and upload_spec_to_cloud with a visitor-supplied Kanaries token (it would overwrite the server's stored credentials).

Residual caveats, documented in the docstring: the shared connection/table namespace and register-then-query timing mean per-user data isolation is not a hard boundary, and expensive queries block every session while they run. The explorer is best suited to trusted users; per-widget isolation requested upstream at Kanaries/pygwalker#747.

Compatibility

  • Works with pygwalker 0.4.9.13+, verified against the current release (0.5.0.1) and main (0.6.0rc0), which renames the widget class (_WalkerWidget → WalkerAnyWidget) and moves some internals; detection and hardening handle both.
  • Polars frames are accepted when the installed pygwalker can read them: every release so far trips over dtype groups polars 1.0 removed, but the fix is merged upstream and due in 0.6 — the renderer probes for it, so polars starts working automatically when a fixed pygwalker is installed, and gives a df.to_pandas() hint until then.
  • No comm changes needed; PyGWalker's anywidget API works over the existing custom-message transport.

Notes

  • Privacy: PyGWalker's update check and anonymous usage events are on by default (Python process and browser). Call pygwalker.GlobalVarManager.set_privacy("offline") to disable both; the docstring and example mention this.
  • The widget has no interaction traits, so there's nothing to reactive_read(); the browser→Python round trip is tested by counting pyg_request messages.
  • The pygwalker extra lists duckdb and sqlglot explicitly since apps and shinywidgets import them directly.

Verification

  • 43 unit tests pass against both pygwalker 0.5.0.1 and main (0.6.0rc0), plus pyright and format checks.
  • 7 Playwright tests (renders cleanly, fill measurement, explicit heights, Python round trip, re-render lifecycle) pass, confirming the SQL validation doesn't break legitimate explorer traffic.
  • Example app smoke-tested with a dataset switch.

Known limitations

  • The Playwright tests need network access for leaflet.css from unpkg (subresource integrity hash, can't be stubbed). Banner images are stubbed.

This comment was marked as resolved.

This comment was marked as resolved.

cpsievert added a commit that referenced this pull request Oct 26, 2026
PyGWalker runs the SQL its browser sends on DuckDB's process-wide default
connection, so a visitor could read or write files on the server unless the
app locked that connection down -- and nothing enforced it. Rendering now
checks the connection's `enable_external_access`/`lock_configuration`, and
raises with the lockdown snippet unless external access is off and the
configuration is locked.

PyGWalker's `upload_spec_to_cloud` request also takes a Kanaries token from
the browser and writes it to `set_config()` and `GlobalVarManager`, i.e. any
visitor could overwrite the server's stored credentials. That request now
refuses a visitor-supplied token (still delegating when the token comes from
the app), and `open_in_desktop`, which shells out to the server's `gw://`
handler, is rejected outright. Both apply to widgets from
`pygwalker.api.anywidget.walk()` too, and rendering fails closed if PyGWalker's
request handlers can't be found.
cpsievert added a commit that referenced this pull request Oct 26, 2026
`render_pygwalker` rejected every polars DataFrame because pygwalker's polars
parser builds field specs from dtype groups that polars 1.0 removed
(`pl.NUMERIC_DTYPES` and friends). Upstream fixed that on main
(Kanaries/pygwalker@5055d702) but hasn't released it, so a blanket rejection
would keep blocking polars after a release ships (and blocks polars < 1.0 users
today).

Decide from the installed packages instead: old polars still has the groups,
and a fixed pygwalker exposes its `_is_numeric_dtype` helper. Unknown answers
count as supported, since this is a friendlier error for a broken combination,
not a guarantee.
cpsievert added a commit that referenced this pull request Oct 27, 2026
render_pygwalker (#289) reads the same two settings (via a
duckdb_settings() helper) to enforce its lockdown, just with the
opposite policy: it runs browser SQL on DuckDB's process-wide default
connection, which shinywidgets can't lock down on the app's behalf, so
it fails closed and requires the app to do it. Adopt its
duckdb_settings() here and add a generic restrict_duckdb_connection()
next to it, so the SET idiom lives in one place once both PRs land.
cpsievert added a commit that referenced this pull request Oct 27, 2026
Stacks this PR on render-embedding-atlas so both DuckDB-backed renderers
share the lockdown mechanics: duckdb_settings() now lives once (next to
restrict_duckdb_connection(), which render_embedding_atlas uses) and this
branch's duplicate is dropped. require_duckdb_lockdown() keeps its
fail-closed policy for PyGWalker's process-wide default connection.

Conflicts: CHANGELOG.md (keep both entries) and the TYPE_CHECKING block
in _render_widget.py (keep both widget imports).
@cpsievert
cpsievert changed the base branch from main to render-embedding-atlas October 27, 2026 21:07
@cpsievert
cpsievert added this pull request to stack #350 October 27, 2026 21:09
@cpsievert
cpsievert marked this pull request as ready for review October 27, 2026 21:10
Renders PyGWalker's drag-and-drop explorer as a Shiny output. Fails
closed unless DuckDB is locked down, validates visitor SQL to single
SELECTs of the widget's own data, and rejects open_in_desktop and
visitor-token upload_spec_to_cloud requests.

Part of #278. Closes as a linear rebase of #289 onto render-embedding-atlas.
@cpsievert
cpsievert merged commit d9f4742 into main Oct 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants