Merge pull request #4095 from plotly/shared-storage-opt-in

Make shared storage opt-in, move msgspec to extras

Author: T4rk1n

.ai/ARCHITECTURE.md

@@ -727,7 +727,7 @@ Special handling for Colab:
 - `prevent_initial_callbacks` - Skip callbacks on load (default: `False`)
 - `background_callback_manager` - DiskcacheManager or CeleryManager
 - `on_error` - Global callback error handler
-- `shared_storage` - Cross-process state + pub/sub backend (default: `LocalSharedStorage`). Pass `None` to disable, or a `BaseSharedStorage` subclass/instance to swap. See [Shared Storage](#shared-storage).
+- `shared_storage` - Cross-process state + pub/sub backend (default: off). Pass `LocalSharedStorage` to enable, or a `BaseSharedStorage` subclass/instance to swap. See [Shared Storage](#shared-storage).
 
 **WebSocket Callbacks:**
 - `websocket_callbacks` - Enable WebSocket for all callbacks (default: `False`). Requires FastAPI backend.
@@ -820,7 +820,9 @@ channel. Unlike `dcc.Store` (which lives in the browser), shared storage lives
 on the server and lets callbacks running in different workers see the same keys
 and topics without standing up an external service like Redis.
 
-Enabled by default on every app. It starts lazily on first use, so it costs
+Opt-in: off unless the app passes `shared_storage=` or `DASH_SHARED_STORAGE`
+is set, so existing deployments do not grow a socket owner or a streaming
+downlink on upgrade. Once enabled it starts lazily on first use, so it costs
 nothing until touched.
 
 ### Accessing It
@@ -836,13 +838,21 @@ def handler(n):
     return store.get("clicks", 0)
 
 # Or off the app directly:
-app = dash.Dash()
+app = dash.Dash(shared_storage=dash.LocalSharedStorage)
 app.shared_storage.set("key", {"any": "json-compatible value"})
 ```
 
 Values (and published messages) must be **JSON-compatible** — dict / list / str
 / int / float / bool / None — the same constraint as `dcc.Store` and callback
-outputs. Messages are encoded with `msgspec` (msgpack), a hard dependency.
+outputs. Messages are encoded with `msgspec` (msgpack) from the
+`dash[shared_storage]` extra (also pulled in by `dash[diskcache]` and
+`dash[redis]`). Without it, `LocalSharedStorage` in `"memory"` mode still works
+and its socket transport falls back to stdlib JSON (both ends run the same
+install, so they always agree). JSON turns non-string dict keys into strings and
+cannot carry `bytes`, so keep values strictly JSON-shaped. Everything that
+writes encoded bytes somewhere lasting (the persistent modes, Diskcache, Redis)
+raises an `ImportError` naming the extra that brings it instead, so a stored
+format never depends on what happens to be installed.
 
 ### API
 
@@ -895,7 +905,7 @@ more than one container/pod (see [Deployment Topology](#deployment-topology-and-
 
 | Backend | Backing store | Reaches | Extra |
 |---------|---------------|---------|-------|
-| `LocalSharedStorage` *(default)* | in-memory, owner-elected socket | one **container** (all its workers) | none |
+| `LocalSharedStorage` | in-memory, owner-elected socket | one **container** (all its workers) | none (`dash[shared_storage]` for the persistent modes) |
 | `DiskcacheSharedStorage` | a `diskcache.Cache` | one **host** (all processes sharing the dir) | `dash[diskcache]` |
 | `RedisSharedStorage` | Redis (Streams for pub/sub) | **any** process/container/pod | `dash[redis]` |
 
@@ -909,14 +919,15 @@ from dash._shared_storage import (
     LocalSharedStorage, DiskcacheSharedStorage, RedisSharedStorage,
 )
 
-Dash(shared_storage=LocalSharedStorage)                       # default
+Dash()                                                        # disabled (default)
+Dash(shared_storage=LocalSharedStorage)
 Dash(shared_storage=DiskcacheSharedStorage(directory="/tmp/ss"))
 Dash(shared_storage=RedisSharedStorage(url="redis://host:6379"))
-Dash(shared_storage=None)                                     # disabled
 ```
 
 When disabled, accessing `app.shared_storage` / `dash.ctx.shared_storage`
-raises `SharedStorageError`.
+raises `SharedStorageError`, and streaming callbacks still work, each on its own
+connection instead of the multiplexed downlink.
 
 **`LocalSharedStorage`** — in-memory, no external service. On first use every
 worker races to become the single **owner** for the machine by binding a stable
@@ -993,7 +1004,7 @@ subscriptions likewise poll without blocking and sleep on the loop.
 
 ### Deployment Topology and Shared Storage
 
-The default `LocalSharedStorage` is **per-container**: its socket/loopback
+`LocalSharedStorage` is **per-container**: its socket/loopback
 election only reaches processes in the same network + filesystem namespace.
 
 - **Single process / single pod** (e.g. Plotly Cloud apps): fine — one owner,
@@ -1009,16 +1020,16 @@ election only reaches processes in the same network + filesystem namespace.
 A hosting platform can switch the backend without editing the app through
 `DASH_SHARED_STORAGE` (`_shared_storage/_env.py`), read only when the app did
 not pass `shared_storage=` (the default is a sentinel, so an explicit argument,
-`None` included, always wins): `local`, `none`, `diskcache:///abs/path`, or a
-`redis://` / `rediss://` URL. A `key_prefix` query parameter on the Redis URL
-(`redis://host/0?key_prefix=myapp`) sets `RedisSharedStorage(key_prefix=)` and
-is stripped before the URL reaches redis-py; the diskcache path is
-percent-decoded. `cluster://` is reserved and raises; anything else raises
-`InvalidConfig` at construction, without echoing the value (it can hold
-credentials). The value becomes a zero-argument
-factory, so nothing is built or connected until `app.shared_storage` is first
-read, but a missing extra (`dash[redis]`, `dash[diskcache]`) fails at
-construction with the backend's own `ImportError`.
+`None` included, always wins): `local`, `none` (same as unset or empty),
+`diskcache:///abs/path`, or a `redis://` / `rediss://` URL. A `key_prefix` query
+parameter on the Redis URL (`redis://host/0?key_prefix=myapp`) sets
+`RedisSharedStorage(key_prefix=)` and is stripped before the URL reaches
+redis-py; the diskcache path is percent-decoded. `cluster://` is reserved and
+raises; anything else raises `InvalidConfig` at construction, without echoing
+the value (it can hold credentials). The value becomes a zero-argument factory,
+so nothing is built or connected until `app.shared_storage` is first read, but a
+missing extra (`dash[redis]`, `dash[diskcache]`, `dash[shared_storage]`) fails
+at construction with the backend's own `ImportError`.
 
 ### Custom and Out-of-Tree Backends
 
@@ -1039,12 +1050,13 @@ and returns a `Subscription` (`__iter__` / `__aiter__` / `close`) that raises
 top-level `dash`; the poll-loop helpers (`PollResult`, the polling subscription)
 are private and not part of the contract.
 
-Core only ships backends whose dependency is already a Dash extra (`msgspec`
-base; `dash[diskcache]`; `dash[redis]`). Anything needing a heavier dependency
-belongs **out of tree** behind this same interface — e.g. a Postgres backend
-(`LISTEN`/`NOTIFY` for push pub/sub + a table for KV and replay) lives in its own
-package so a `psycopg` connection is never pulled into Dash core. It plugs in the
-same way as a built-in: `Dash(shared_storage=PostgresSharedStorage(...))`.
+Core only ships backends whose dependency is already a Dash extra
+(`dash[shared_storage]`; `dash[diskcache]`; `dash[redis]`). Anything needing a
+heavier dependency belongs **out of tree** behind this same interface — e.g. a
+Postgres backend (`LISTEN`/`NOTIFY` for push pub/sub + a table for KV and
+replay) lives in its own package so a `psycopg` connection is never pulled into
+Dash core. It plugs in the same way as a built-in:
+`Dash(shared_storage=PostgresSharedStorage(...))`.
 
 ### Module Layout
 
@@ -1412,7 +1424,7 @@ with an empty result on the terminal frame.
 
 ### Multiplexed downlink and the stream SharedWorker
 
-When the app has a shared-storage backend (the default `LocalSharedStorage`),
+When the app has a shared-storage backend (opt-in, see Shared Storage),
 HTTP streams do not each hold their own response. The callback's POST carries
 `streamConnection: {requestId}` and returns a fast ack; a *pump*
 (`dash/_stream_hub.py`) drives the generator as an asyncio task and publishes

CHANGELOG.md

@@ -5,12 +5,13 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 ## [Unreleased]
 
 ### Added
+- [#4095](https://github.com/plotly/dash/pull/4095) Make shared storage and the multiplexed streaming downlink opt-in (`shared_storage=` or `DASH_SHARED_STORAGE`), and move `msgspec` to a new `dash[shared_storage]` extra for the persistent `LocalSharedStorage` modes (`dash[diskcache]` and `dash[redis]` include it).
 - [#4026](https://github.com/plotly/dash/pull/4026) Add `DASH_SECRET_KEY` so a hosting platform can give every worker and pod the same signing secret for page, background and stream tokens without editing the app (`server.secret_key` still wins and is not changed).
 - [#4026](https://github.com/plotly/dash/pull/4026) Add `DASH_SHARED_STORAGE` to pick the shared-storage backend when the app does not pass `shared_storage=`: `local`, `none`, `diskcache:///abs/path`, or a `redis://` / `rediss://` URL.
 - [#4024](https://github.com/plotly/dash/pull/4024) Add support for social media meta tags in regular Dash apps using `app.title`, `app.description`, and `app.image`.
 - [#3976](https://github.com/plotly/dash/pull/3976) Add a new `scrollToTop` prop to `dcc.Link` to control whether the page scrolls to the top after client-side navigation. It defaults to `True` to preserve the existing behavior. Fixes [#3974](https://github.com/plotly/dash/issues/3974).
 - [#3947](https://github.com/plotly/dash/pull/3947) Make `plotly-cloud` a default install dependency of Dash instead of an optional extra, so the `plotly` CLI and Dash's cloud integration work out of the box. The `dash[cloud]` extra is kept for backward compatibility.
-- [#3930](https://github.com/plotly/dash/pull/3930) Add shared storage: a backend-agnostic cross-process state manager (key/value with optional TTL, plus ordered replayable pub/sub) on every app via `dash.ctx.shared_storage`, started lazily and disabled with `shared_storage=None`. Ships `LocalSharedStorage` (default, in-memory with optional disk persistence), `DiskcacheSharedStorage`, and `RedisSharedStorage` for horizontally-scaled deployments; see `.ai/ARCHITECTURE.md`.
+- [#3930](https://github.com/plotly/dash/pull/3930) Add shared storage: a backend-agnostic cross-process state manager (key/value with optional TTL, plus ordered replayable pub/sub) via `dash.ctx.shared_storage`, started lazily. Ships `LocalSharedStorage` (in-memory with optional disk persistence), `DiskcacheSharedStorage`, and `RedisSharedStorage` for horizontally-scaled deployments; see `.ai/ARCHITECTURE.md`.
 - [#3931](https://github.com/plotly/dash/pull/3931) Add streaming callbacks: an `async def` generator callback streams its yields to the browser as they are produced (`dash.Patch` yields apply incrementally). A browser's streams share one SharedWorker-hosted connection so they do not count against the per-host connection limit; closing a tab cancels its streams.
 - [#3977](https://github.com/plotly/dash/pull/3977) Add partial WebSocket prop reads with `get_prop(..., path=...)`. Closes [#3975](https://github.com/plotly/dash/issues/3975).
 - [#3765](https://github.com/plotly/dash/pull/3765) Add opt-in partial pattern matching for callback `Input`, `Output`, and `State` via `partial_pattern=True`, so dictionary ID patterns can match component IDs with extra keys and combine with `ALL`/`MATCH` wildcards. Fixes [#3764](https://github.com/plotly/dash/issues/3764).

dash/_callback_context.py

@@ -216,7 +216,7 @@ def shared_storage(self):
 
         A backend-agnostic key/value + publish/subscribe store shared across
         worker processes -- for cross-callback or session state without an
-        external service. Requires ``Dash(shared_storage=...)`` (on by default).
+        external service. Requires ``Dash(shared_storage=...)`` (off by default).
         """
         return get_app().shared_storage
 

dash/_shared_storage/__init__.py

@@ -2,7 +2,7 @@
 
 Exposes the ``BaseSharedStorage`` interface and its backends:
 
-- ``LocalSharedStorage`` (default): in-memory, owner-elected -- one container.
+- ``LocalSharedStorage``: in-memory, owner-elected -- one container.
 - ``DiskcacheSharedStorage``: on a ``diskcache.Cache`` -- one machine, not pods.
 - ``RedisSharedStorage``: on Redis -- across processes and containers.
 """

dash/_shared_storage/_codec.py

@@ -1,27 +1,53 @@
-"""Codec for the shared-storage socket transport.
+"""Codec for the shared-storage transport and the persistent backends.
 
-Uses ``msgspec`` (msgpack: fast, compact), a hard dependency of Dash. The
-format is data-only, so bytes read off the socket cannot execute code.
+Uses ``msgspec`` (msgpack: fast, compact) from the ``dash[shared_storage]``
+extra, also part of ``dash[diskcache]`` and ``dash[redis]``. Without it, only the in-memory transport works and it falls back to
+stdlib JSON; the backends that write encoded bytes somewhere lasting (disk
+persistence, diskcache, Redis) call :func:`require_msgspec` so the stored
+format never depends on what happens to be installed. Both formats are
+data-only, so bytes read off the socket cannot execute code.
 
 Payloads must be JSON-compatible (dict / list / str / int / float / bool /
 None) -- the same constraint as ``dcc.Store`` and callback outputs.
 """
 
+import json
 from typing import Any
 
-import msgspec
+try:
+    import msgspec
+except ImportError:
+    msgspec = None  # type: ignore[assignment]
 
-# msgspec raises this on malformed input. It subclasses ``ValueError`` in
-# current msgspec but not in every version, so callers should catch it by name.
-DecodeError = msgspec.DecodeError
+HAS_MSGSPEC = msgspec is not None
 
-_encoder = msgspec.msgpack.Encoder()
-_decoder = msgspec.msgpack.Decoder()
+if msgspec is not None:
+    # msgspec raises this on malformed input. It subclasses ``ValueError`` in
+    # current msgspec but not in every version, so callers should catch it by
+    # name.
+    DecodeError: Any = msgspec.DecodeError
+    _encoder = msgspec.msgpack.Encoder()
+    _decoder = msgspec.msgpack.Decoder()
 
+    def encode(obj: Any) -> bytes:
+        return _encoder.encode(obj)
 
-def encode(obj: Any) -> bytes:
-    return _encoder.encode(obj)
+    def decode(data: bytes) -> Any:
+        return _decoder.decode(data)
 
+else:
+    DecodeError = json.JSONDecodeError
 
-def decode(data: bytes) -> Any:
-    return _decoder.decode(data)
+    def encode(obj: Any) -> bytes:
+        return json.dumps(obj, separators=(",", ":")).encode("utf-8")
+
+    def decode(data: bytes) -> Any:
+        return json.loads(data)
+
+
+def require_msgspec(what: str, extra: str = "shared_storage") -> None:
+    if not HAS_MSGSPEC:
+        raise ImportError(
+            f"{what} requires the {extra} extra:\n\n"
+            f'    $ pip install "dash[{extra}]"\n'
+        )

dash/_shared_storage/_env.py

@@ -49,10 +49,10 @@ def storage_from_env(value: Optional[str]) -> Any:
     """
     raw = (value or "").strip()
     lowered = raw.lower()
-    if lowered in ("", "local"):
-        return LocalSharedStorage
-    if lowered == "none":
+    if lowered in ("", "none"):
         return None
+    if lowered == "local":
+        return LocalSharedStorage
 
     try:
         scheme = urlparse(raw).scheme.lower()

dash/_shared_storage/_persistence.py

@@ -4,8 +4,8 @@
 keys are sharded across fixed-size chunk files via a key->chunk index, so a
 single mutation rewrites only the affected chunk rather than the whole store
 (the design is adapted from raposa's ``BinaryStorage``). Serialization is
-msgspec msgpack via :mod:`._codec`; each chunk write is atomic (temp file +
-``os.replace``).
+msgspec msgpack via :mod:`._codec` (the ``dash[shared_storage]`` extra); each
+chunk write is atomic (temp file + ``os.replace``).
 
 Two modes drive *when* the store is written:
 

dash/_shared_storage/_transport.py

@@ -3,9 +3,9 @@
 A single owner process serves one :class:`StoreEngine` over a stream socket
 (AF_UNIX where available, TCP loopback otherwise). Messages are length-prefixed:
 a 4-byte big-endian length followed by a body encoded by ``_codec`` (msgspec
-msgpack). The codec is data-only -- the socket is
-localhost-only, but deserializing untrusted bytes off a socket is a
-remote-code-execution hazard, so the transport carries only data. The channel is
+msgpack, or JSON without the ``dash[shared_storage]`` extra). The codec is
+data-only -- the socket is localhost-only, but deserializing untrusted bytes off
+a socket is a remote-code-execution hazard, so the transport carries only data. The channel is
 additionally gated by a per-owner random token; only same-host clients that read
 the owner's advertisement file (written 0600) can attach.
 

dash/_shared_storage/diskcache.py

@@ -20,7 +20,7 @@
 import time
 from typing import Any, Optional
 
-from ._codec import decode, encode
+from ._codec import decode, encode, require_msgspec
 from ._engine import DEFAULT_BUFFER, PollResult
 from ._polling import PollingSubscription
 from .base import BaseSharedStorage, Subscription, check_topic_ttl
@@ -37,6 +37,7 @@
 
 
 def _require_diskcache():
+    require_msgspec("DiskcacheSharedStorage", "diskcache")
     try:
         import diskcache  # type: ignore[import-not-found,import-untyped] # pylint: disable=import-outside-toplevel
 

dash/_shared_storage/local.py

@@ -25,6 +25,7 @@
 import time
 from typing import Any, Optional, Tuple
 
+from ._codec import require_msgspec
 from ._engine import DEFAULT_BUFFER, PollResult, StoreEngine
 from ._persistence import _Persistence, default_store_dir
 from ._transport import (
@@ -477,6 +478,7 @@ class LocalSharedStorage(BaseSharedStorage):
       ``flush_interval`` seconds and on clean exit, and recovered on start; up to
       one interval of writes may be lost on an unclean crash.
 
+    The persistent modes need the ``dash[shared_storage]`` extra (msgspec).
     ``path`` is the on-disk store directory for the persistent modes; it defaults
     to a per-namespace folder under the user cache directory. ``flush_interval``
     (seconds) only applies to ``"persist-reset"``.
@@ -501,6 +503,8 @@ def __init__(
             raise SharedStorageError(
                 f"flush_interval must be positive, got {flush_interval!r}"
             )
+        if mode != "memory":
+            require_msgspec(f"LocalSharedStorage(mode={mode!r})")
         ns = namespace or _default_namespace()
         if mode != "memory" and path is None:
             path = default_store_dir(ns)