Fix 1983

.audit/oberstet_fix_1983.md

@@ -0,0 +1,8 @@
+- [ ] I did **not** use any AI-assistance tools to help create this pull request.
+- [x] I **did** use AI-assistance tools to *help* create this pull request.
+- [x] I have read, understood and followed the project's AI_POLICY.md when creating code, documentation etc. for this pull request.
+
+Submitted by: @oberstet
+Date: 2026-10-13
+Related issue(s): #1983
+Branch: oberstet:fix_1983

docs/index.rst

@@ -44,6 +44,7 @@ Contents
    overview
    installation
    wheels-inventory
+   memory-footprint
    getting-started
    programming-guide/index
    architecture/statically-typed-region-lifecycle-python.md

docs/memory-footprint.rst

@@ -0,0 +1,144 @@
+.. _memory-footprint:
+
+Memory Footprint
+================
+
+This page shows how much memory a Python process uses after importing autobahn,
+compared with the frameworks and libraries it builds on, for CPython and PyPy. It
+helps to see what a given use of autobahn costs, and it documents how that cost
+changed across releases.
+
+Background
+----------
+
+Since 25.12.1, autobahn includes its own copy of the
+`FlatBuffers <https://flatbuffers.dev/>`_ runtime, which it needs for WAMP
+FlatBuffers serialization (before that, ``flatbuffers`` was an optional dependency,
+the ``serialization`` extra). That runtime imports **numpy** at module level whenever
+numpy is installed. Up to 26.7.1, autobahn imported the runtime eagerly, so every
+process that imported autobahn also loaded numpy - including processes that use
+only WebSocket, not WAMP, such as Django Channels applications served by
+`Daphne`_ [#fn-daphne]_. See `issue #1983 <https://github.com/crossbario/autobahn-python/issues/1983>`_.
+
+Since then, the goals are:
+
+1. The autobahn package, and the wheels we build, include the vendored FlatBuffers
+   runtime, including reflection.
+2. For an application that depends on both autobahn and zlmdb, both of which vendor
+   FlatBuffers, such as Crossbar.io, the two vendored versions are exactly the same:
+   :func:`autobahn.check_zlmdb_flatbuffers_version_in_sync` checks that.
+3. A user of autobahn who uses **WebSocket only, not WAMP**, with either Twisted or
+   asyncio, loads no FlatBuffers code and no numpy.
+
+autobahn now loads the FlatBuffers runtime only when a FlatBuffers message is first
+built or parsed.
+
+What is measured
+----------------
+
+Each row is one import, run in a fresh interpreter. The value is the peak resident
+set size of that process right after the import, in MB (MiB): ``VmHWM`` from
+``/proc/self/status`` on Linux. It is the memory the import costs at least, not the
+memory a running application uses under load.
+
+(``ru_maxrss`` from :func:`resource.getrusage` would be the obvious choice, but on
+Linux it carries over the peak memory of the process that started the measured one,
+so for a child of a Python process it never drops below that parent's size.)
+
+* Every venv has numpy installed, so the tables show whether an import loads it.
+  A value marked **†** means numpy was loaded by that import.
+* Each import runs once as a warm-up (which may compile ``.pyc`` files) and is then
+  measured 5 times; the table shows the median.
+* Each autobahn version is measured in its own fresh venv, together with Twisted,
+  cryptography, numpy and Daphne. The rows that do not involve autobahn are measured
+  in every venv, so their spread across the columns shows the measurement noise.
+* Crossbar.io [#fn-crossbar]_ is measured in a separate venv per autobahn version, with
+  crossbar |crossbar_version| from PyPI. No crossbar release supports autobahn 25.10.2,
+  hence "n/a".
+* Columns: **25.10.2** is the last release with FlatBuffers as an optional
+  dependency, measured *without* that extra (as Daphne installs it). **26.7.1** is the
+  latest release. The last column is the current development version.
+
+.. |crossbar_version| replace:: 26.7.1
+
+CPython 3.14
+------------
+
+.. include:: memory-footprint/cpy314.inc
+
+.. include:: memory-footprint/cpy314-versions.inc
+
+PyPy 3.11
+---------
+
+PyPy is a different runtime (a tracing JIT compiler with its own garbage collector),
+so compare these values with each other, not with the CPython table. On PyPy, the
+venvs use numpy 2.4.x, the last numpy release with PyPy wheels.
+
+.. include:: memory-footprint/pypy311.inc
+
+.. include:: memory-footprint/pypy311-versions.inc
+
+Findings
+--------
+
+* **This change:** every row that imports autobahn drops by about 13-17 MB on
+  CPython, and by about 18-24 MB on PyPy, and none of them loads numpy any more. For
+  example, ``from daphne.server import Server`` drops from 64.6 MB to 51.6 MB on CPython
+  3.14, and ``import autobahn`` alone is back to the size of the bare interpreter.
+* **Compared with 25.10.2:** the WebSocket and WAMP rows remain about 1.3-1.6 MB above
+  25.10.2 on CPython (about 0.4-0.8 MB on PyPy). In 25.10.2 without the ``serialization``
+  extra, cbor2 was not installed and the WAMP message module skipped that import. Since
+  26.x, cbor2 is a regular dependency.
+* **WebSocket only:** with Twisted, most of what remains is Twisted itself; with
+  asyncio, the asyncio standard library.
+* **Twisted and cryptography:** ``import autobahn.twisted.websocket`` still loads the
+  native ``cryptography`` bindings (about 8 MB on CPython), because
+  ``autobahn/twisted/__init__.py`` imports ``autobahn.twisted.wamp``, which imports
+  ``autobahn.wamp.auth``. A pure WebSocket user does not need them. ``import cryptography``
+  alone costs next to nothing; the cost comes with the native bindings
+  (``from cryptography.hazmat.primitives import hashes``).
+* **WAMP:** WAMP users still load the FlatBuffers runtime (and numpy, if installed)
+  as soon as they import ``autobahn.wamp.serializer``, also when they use another
+  serializer.
+* **Crossbar.io** loads numpy itself, so its router worker loads it regardless of
+  this change; on CPython, its footprint is unchanged. On PyPy, the crossbar row comes
+  out about 16 MB lower with this change, in repeated measurements, but we have not
+  analysed why. Two things to keep in mind: on PyPy, the samples of a single import
+  can spread by about 10 MB from run to run (garbage collection and JIT timing), and
+  the two crossbar venvs differ in more than autobahn: on PyPy, autobahn 26.7.1 installs
+  cbor2 6.x, while the development version installs cbor2 5.9.0 (see the versions
+  below the PyPy table).
+
+Reproducing
+-----------
+
+The scripts are in `scripts/memory-footprint/
+<https://github.com/crossbario/autobahn-python/tree/master/scripts/memory-footprint>`_.
+They create throwaway venvs with `uv <https://docs.astral.sh/uv/>`_, measure, and
+regenerate the tables on this page:
+
+.. code-block:: console
+
+    just memory-footprint cpy314
+    just memory-footprint pypy311
+
+``measure.py`` alone measures the venv it runs in, e.g. for your own environment:
+
+.. code-block:: console
+
+    python scripts/memory-footprint/measure.py --variant mine
+
+.. rubric:: Footnotes
+
+.. [#fn-daphne] `Daphne <https://pypi.org/project/daphne/>`_
+   (`repository <https://github.com/django/daphne>`_), the ASGI server of Django
+   Channels. Its `server <https://github.com/django/daphne/blob/main/daphne/server.py>`_
+   and `WebSocket protocol <https://github.com/django/daphne/blob/main/daphne/ws_protocol.py>`_
+   use ``autobahn.twisted.websocket``. Row: ``from daphne.server import Server``.
+
+.. [#fn-crossbar] `Crossbar.io <https://github.com/crossbario/crossbar>`_, the WAMP
+   router. Row: ``from crossbar.edge.worker.router import ExtRouterController``, the
+   `router worker <https://github.com/crossbario/crossbar/blob/master/src/crossbar/edge/worker/router.py>`_.
+
+.. _Daphne: https://github.com/django/daphne

docs/memory-footprint/cpy314-versions.inc

@@ -0,0 +1,7 @@
+* **25.10.2** (venv): autobahn 25.10.2, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.5.4, daphne 4.2.3
+* **26.7.1** (venv): autobahn 26.7.1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.5.4, cbor2 6.1.5, daphne 4.2.3
+* **26.7.1** (crossbar venv): autobahn 26.7.1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.5.4, cbor2 6.1.5, crossbar 26.7.1, zlmdb 26.7.1
+* **fix_1983** (venv): autobahn 26.10.1.dev1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.5.4, cbor2 6.1.5, daphne 4.2.3
+* **fix_1983** (crossbar venv): autobahn 26.10.1.dev1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.5.4, cbor2 6.1.5, crossbar 26.7.1, zlmdb 26.7.1
+
+Interpreter: ``CPython 3.14.7`` (3.14.7 (main, Sep 1 2026, 14:18:09) [Clang 22.1.3 ]); Linux 6.12.95+deb12-amd64 x86_64, median of 5 runs.

docs/memory-footprint/cpy314.inc

@@ -0,0 +1,89 @@
+.. list-table::
+   :header-rows: 1
+   :widths: 12 46 14 14 14
+
+   * - group
+     - import
+     - 25.10.2
+     - 26.7.1
+     - fix_1983
+   * - interpreter
+     - ``bare interpreter``
+     - 11.9 MB
+     - 11.9 MB
+     - 12.0 MB
+   * - cryptography
+     - ``import cryptography``
+     - 12.0 MB
+     - 12.0 MB
+     - 11.9 MB
+   * -
+     - ``from cryptography.hazmat.primitives import hashes``
+     - 19.9 MB
+     - 19.9 MB
+     - 19.8 MB
+   * - numpy
+     - ``import numpy``
+     - 28.4 MB †
+     - 28.4 MB †
+     - 28.4 MB †
+   * - asyncio
+     - ``import asyncio``
+     - 18.1 MB
+     - 18.2 MB
+     - 18.2 MB
+   * - twisted
+     - ``from twisted.internet import reactor``
+     - 43.3 MB
+     - 43.3 MB
+     - 43.4 MB
+   * - txaio
+     - ``import txaio``
+     - 12.0 MB
+     - 12.1 MB
+     - 12.1 MB
+   * -
+     - ``import txaio; txaio.use_asyncio()``
+     - 18.3 MB
+     - 18.3 MB
+     - 18.3 MB
+   * -
+     - ``import txaio; txaio.use_twisted()``
+     - 26.5 MB
+     - 26.5 MB
+     - 26.5 MB
+   * - autobahn
+     - ``import autobahn``
+     - 12.1 MB
+     - 28.8 MB †
+     - 12.0 MB
+   * -
+     - ``txaio.use_asyncio(); import autobahn.wamp``
+     - 31.9 MB
+     - 47.6 MB †
+     - 33.2 MB
+   * -
+     - ``txaio.use_twisted(); import autobahn.wamp``
+     - 38.4 MB
+     - 52.9 MB †
+     - 39.8 MB
+   * -
+     - ``import autobahn.asyncio.websocket``
+     - 36.3 MB
+     - 51.1 MB †
+     - 37.8 MB
+   * -
+     - ``import autobahn.twisted.websocket``
+     - 48.9 MB
+     - 63.2 MB †
+     - 50.3 MB
+   * - daphne
+     - ``from daphne.server import Server``
+     - 50.3 MB
+     - 64.6 MB †
+     - 51.6 MB
+   * - crossbar
+     - ``from crossbar.edge.worker.router import ExtRouterController``
+     - n/a
+     - 144.7 MB †
+     - 144.7 MB †

docs/memory-footprint/pypy311-versions.inc

@@ -0,0 +1,7 @@
+* **25.10.2** (venv): autobahn 25.10.2, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.4.6, daphne 4.2.3
+* **26.7.1** (venv): autobahn 26.7.1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.4.6, cbor2 6.1.5, daphne 4.2.3
+* **26.7.1** (crossbar venv): autobahn 26.7.1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.4.6, cbor2 6.1.5, crossbar 26.7.1, zlmdb 26.7.1
+* **fix_1983** (venv): autobahn 26.10.1.dev1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.4.6, cbor2 5.9.0, daphne 4.2.3
+* **fix_1983** (crossbar venv): autobahn 26.10.1.dev1, txaio 26.6.1, twisted 26.4.0, cryptography 50.0.2, numpy 2.4.6, cbor2 5.9.0, crossbar 26.7.1, zlmdb 26.7.1
+
+Interpreter: ``PyPy 3.11.15`` (3.11.15 (194f9f44b505, May 25 2026, 19:34:11) [PyPy 7.3.23 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]); Linux 6.12.95+deb12-amd64 x86_64, median of 5 runs.

docs/memory-footprint/pypy311.inc

@@ -0,0 +1,89 @@
+.. list-table::
+   :header-rows: 1
+   :widths: 12 46 14 14 14
+
+   * - group
+     - import
+     - 25.10.2
+     - 26.7.1
+     - fix_1983
+   * - interpreter
+     - ``bare interpreter``
+     - 64.4 MB
+     - 64.4 MB
+     - 64.4 MB
+   * - cryptography
+     - ``import cryptography``
+     - 64.3 MB
+     - 64.4 MB
+     - 64.4 MB
+   * -
+     - ``from cryptography.hazmat.primitives import hashes``
+     - 72.1 MB
+     - 72.0 MB
+     - 72.0 MB
+   * - numpy
+     - ``import numpy``
+     - 88.1 MB †
+     - 88.1 MB †
+     - 88.2 MB †
+   * - asyncio
+     - ``import asyncio``
+     - 78.7 MB
+     - 78.9 MB
+     - 78.8 MB
+   * - twisted
+     - ``from twisted.internet import reactor``
+     - 106.1 MB
+     - 106.1 MB
+     - 106.1 MB
+   * - txaio
+     - ``import txaio``
+     - 64.4 MB
+     - 64.4 MB
+     - 64.2 MB
+   * -
+     - ``import txaio; txaio.use_asyncio()``
+     - 79.0 MB
+     - 79.0 MB
+     - 79.1 MB
+   * -
+     - ``import txaio; txaio.use_twisted()``
+     - 88.0 MB
+     - 87.9 MB
+     - 88.0 MB
+   * - autobahn
+     - ``import autobahn``
+     - 64.4 MB
+     - 87.9 MB †
+     - 64.4 MB
+   * -
+     - ``txaio.use_asyncio(); import autobahn.wamp``
+     - 93.0 MB
+     - 112.4 MB †
+     - 93.7 MB
+   * -
+     - ``txaio.use_twisted(); import autobahn.wamp``
+     - 100.1 MB
+     - 118.8 MB †
+     - 100.5 MB
+   * -
+     - ``import autobahn.asyncio.websocket``
+     - 98.4 MB
+     - 117.2 MB †
+     - 99.0 MB
+   * -
+     - ``import autobahn.twisted.websocket``
+     - 113.2 MB
+     - 133.5 MB †
+     - 113.8 MB
+   * - daphne
+     - ``from daphne.server import Server``
+     - 114.8 MB
+     - 135.8 MB †
+     - 115.6 MB
+   * - crossbar
+     - ``from crossbar.edge.worker.router import ExtRouterController``
+     - n/a
+     - 239.5 MB †
+     - 224.1 MB †

justfile

@@ -1325,7 +1325,8 @@ test-asyncio venv="" use_nvx="": (install-tools venv) (install-dev venv)
 
 # Run the import smoke test - imports core autobahn modules with crypto extras
 # present, guarding against import-time annotation regressions (e.g. #1878) that
-# only surface on CPython < 3.14. (usage: `just test-imports cpy311`)
+# only surface on CPython < 3.14, and checks that WebSocket-only imports load
+# neither FlatBuffers nor numpy (#1983). (usage: `just test-imports cpy311`)
 test-imports venv="": (install-tools venv) (install-dev venv)
     #!/usr/bin/env bash
     set -e
@@ -1337,7 +1338,8 @@ test-imports venv="": (install-tools venv) (install-dev venv)
     fi
     VENV_PYTHON=$(just --quiet _get-venv-python "${VENV_NAME}")
     echo "==> Running import smoke test in ${VENV_NAME}..."
-    ${VENV_PYTHON} -m pytest -v src/autobahn/test/test_import_all.py
+    ${VENV_PYTHON} -m pytest -v src/autobahn/test/test_import_all.py \
+        src/autobahn/test/test_import_websocket_only.py
 
 # Run the import smoke test against the LOWEST versions our direct dependencies
 # allow (uv --resolution lowest-direct), in a separate '<venv>-lowest' venv. This
@@ -1813,6 +1815,11 @@ bump-flatbuffers:
 update-flatbuffers:
     ./scripts/update_flatbuffers.sh
 
+# Measure import memory footprint in throwaway venvs, regenerate docs/memory-footprint.rst tables (usage: `just memory-footprint cpy314`)
+memory-footprint env="cpy314" label="":
+    ./scripts/memory-footprint/run.sh {{ env }} {{ label }}
+    python3 ./scripts/memory-footprint/render.py {{ env }}
+
 # Build wheel only (usage: `just build cpy314`)
 build venv="": (install-build-tools venv)
     #!/usr/bin/env bash

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "autobahn"
-version = "26.9.1.dev2"
+version = "26.10.1.dev1"
 description = "WebSocket client & server library, WAMP real-time framework"
 readme = "README.md"
 requires-python = ">=3.11"