IOPub drops messages without any error once 1000 are queued (common with transport encryption) #1590

Description

The IOPub socket is an XPUB socket with libzmq's default send high water mark of 1000 messages. When 1000 messages wait in the queue to a subscriber, libzmq drops each new message and reports no error. A cell that sends more than 1000 IOPub messages faster than the socket drains them loses the rest, and neither the kernel nor the frontend records it.

Over plain TCP the queue drains fast enough for this to be rare. With CurveZMQ transport encryption (#1515), libzmq encrypts every frame in the kernel's I/O thread, and jupyter_server decrypts it in its own. The pyzmq 27.2.0 Linux wheels build libsodium without optimisation (zeromq/pyzmq#2227), so XPUB to SUB throughput falls from about 57,000 to 23,500 messages per second, and bursts overflow the queue. ipywidgets grids are the common case, because each Button opens three comms (the Button, its Layout and its ButtonStyle).

These trials ran through jupyter_server with JupyterLab's websocket subprotocol, on 8 CPUs, with one fresh kernel per trial. A trial fails if any comm_open, the display_data or the idle status is lost:

In JupyterLab 4.6.5 with jupyterlab_widgets 3.0.17, a lost comm_open shows the ipywidgets error box (widget model not found). For 350 to 500 buttons the end of the burst is lost. The end holds the display_data, so the output stays empty, and in 4 to 10 of 20 trials the idle status too, so the cell stays at [*]. Plain TCP also lost messages on 2 CPUs: 2 of 20 trials at 1000 buttons, against 14 of 20 with CurveZMQ.

Reproduce

pip install ipykernel==7.4.0 jupyter_client==8.10.0 ipywidgets==8.1.9 pyzmq==27.2.0, then:

import queue

from jupyter_client.manager import KernelManager

CODE = """
import ipywidgets as w
display(w.GridBox([w.Button(description=str(i)) for i in range(1000)]))
"""

for encryption in ("disabled", "required"):
    km = KernelManager(kernel_name="python3", transport_encryption=encryption)
    km.start_kernel()
    kc = km.client()
    kc.start_channels()
    kc.wait_for_ready(timeout=60)
    msg_id = kc.execute(CODE)
    opened = 0
    while True:
        try:
            msg = kc.get_iopub_msg(timeout=5)
        except queue.Empty:
            break  # the idle status can be dropped as well
        if msg["parent_header"].get("msg_id") == msg_id:
            opened += msg["msg_type"] == "comm_open"
            if msg["msg_type"] == "status" and msg["content"]["execution_state"] == "idle":
                break
    print(f"transport_encryption={encryption}: {opened} of 3002 comm_open messages received")
    kc.stop_channels()
    km.shutdown_kernel(now=True)

Two runs out of three print:

transport_encryption=disabled: 3002 of 3002 comm_open messages received
transport_encryption=required: 2527 of 3002 comm_open messages received

comm.create_comm() in a loop loses messages the same way, so ipywidgets is not needed: 3000 comms lost 502 messages on average with encryption (9 of 10 trials) and none without (0 of 10).

Cause

The IOPub socket is created and bound with libzmq's defaults:

While it builds 1000 buttons, the IOPub thread sends up to 1130 messages within 20 ms. With encryption, almost nothing leaves the queue during that time: 350, 400 and 500 buttons lost 61 of 1052, 187 of 1202 and 503 of 1502 messages on average. That is the excess over about 1000.

Possible fixes

Two libzmq socket options each removed the loss when set on the IOPub socket. I tested each one with a kernel launcher that sets it, with CurveZMQ on, at 1000 and 3000 buttons, with 20 to 30 trials per case.

• ZMQ_XPUB_NODROP: wait instead of dropping (zmq_setsockopt). When the queue to one subscriber is full, an XPUB socket normally drops the message for that subscriber (zmq_socket). With this option set to 1 (socket.setsockopt(zmq.XPUB_NODROP, 1) in pyzmq), send() waits until the queue has room instead. With zmq.NOBLOCK, it raises zmq.Again.

  With the option on, 0 of 30 trials at 1000 buttons and 0 of 20 at 3000 buttons lost messages. The median time to idle for 1000 buttons was 1.09 s, against 0.99 s over plain TCP.

  The cost is that subscribers stop being independent. One subscriber that stops reading (a suspended jupyter console --existing, for example) now blocks IOPub for every subscriber. In a test with two subscribers, send() blocked after 5000 messages, and the subscriber that was still reading stopped receiving too. A send timeout (ZMQ_SNDTIMEO) would bound the wait. When it expires, ipykernel could log a warning and resend the message with XPUB_NODROP off, so that the message reaches every subscriber with room and is dropped only for the full one. In a test of this fallback, the subscriber that was reading received all 5601 messages.

• ZMQ_SNDHWM: a longer queue (zmq_setsockopt). "High water mark" (HWM) is ZeroMQ's name for the limit on how many messages a socket keeps queued for each peer, here each subscriber (ZeroMQ Guide). libzmq's default is 1000. With SNDHWM=10000, 0 of 20 trials lost messages at either size. A subscriber that stops reading can then hold up to 10,000 queued messages, about 13.5 MB of comm_open messages.

  The option has to be set before bind(). When bind() runs, libzmq copies the socket options into the listener, and each subscriber that connects later gets its queue limit from that copy (stream_listener_base.cpp, session_base.cpp). A later change resizes only the queues of subscribers that are already connected. In an earlier run, the launcher set SNDHWM=10000 after bind() but before any subscriber had connected. Every subscriber still got the default queue of 1000, and 20 of 20 trials failed. pyzmq's Socket.set_hwm warns that new values only take effect for later binds and connects. The libzmq documentation does not: its caution lists ZMQ_SNDHWM among the options that take effect at any time.

The two combine: a larger queue absorbs bursts, and XPUB_NODROP with a timeout covers what remains. Either way, a log line when a send has to wait or drop would make the problem visible. Today nothing records it.

A faster CURVE build lowers the frequency but does not remove the failure. With Ubuntu's libsodium preloaded in both processes, 7 of 30 trials at 1000 buttons still lost messages.

Context

Versions:

• ipykernel 7.4.0, jupyter_client 8.10.0, jupyter_server 2.21.1
• JupyterLab 4.6.5, ipywidgets 8.1.9, jupyterlab_widgets 3.0.17
• pyzmq 27.2.0 (libzmq 4.3.5)
• Python 3.12.13, Linux x86-64

Related issues:

• Widget output rendered before its comm_open is processed fails permanently ("model not found" or stuck "Loading widget...") jupyter-widgets/ipywidgets#4026
• BUG: large CURVE performance penalty on Linux due to unoptimised libsodium build zeromq/pyzmq#2227