Skip to content

Bad pipe messages #1107

Open
Open
Bad pipe messages#1107

Description

@v-jharrison
v-jharrison
opened on Apr 3, 2023

When executing cells in a notebook in Azure Machine learning the following message is output occasionally
Bad pipe message: %s [b'\xc6\x9c\x9b \xf2\xe6\xa7\xb2\xfc\x82\xfa\xd6\xd8\x1d\x8d>\x0c\x08\x00\x00\xa2\xc0\x14\xc0\n\x009\x008\x007\x006\x00\x88\x00\x87\x00\x86\x00\x85\xc0\x19\x00:\x00\x89\xc0\x0f'] Bad pipe message: %s [b'\xa1\x8a\xb0#\x05,\x8d\xaa+L\xfc \x1c\x97\x01'] Bad pipe message: %s [b"\x14\xd3{W-.\x0e}\x9b\xa5&\xc2[\x80=\xad\xda\xba\x00\x00|\xc0,\xc00\x00\xa3\x00\x9f\xcc\xa9\xcc\xa8\xcc\xaa\xc0\xaf\xc0\xad\xc0\xa3\xc0\x9f\xc0]\xc0a\xc0W\xc0S\xc0+\xc0/\x00\xa2\x00\x9e\xc0\xae\xc0\xac\xc0\xa2\xc0\x9e\xc0\\\xc0\xc0V\xc0R\xc0$\xc0(\x00k\x00j\xc0#\xc0'\x00g\x00@\xc0\n\xc0\x14\x009\x008\xc0\t\xc0\x13\x003\x002\x00\x9d\xc0\xa1\xc0\x9d\xc0Q\x00\x9c\xc0\xa0\xc0\x9c\xc0P\x00=\x00<\x005\x00/\x00\x9a\x00\x99\xc0\x07\xc0\x11\x00\x96\x00\x05\x00\xff\x01\x00\x00j\x00\x00\x00\x0e\x00\x0c\x00\x00\t127.0.0.1\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x00\x0c\x00\n\x00\x1d\x00\x17\x00\x1e\x00\x19\x00\x18\x00#\x00\x00\x00\x16\x00\x00\x00\x17\x00\x00\x00\r\x000\x00.\x04\x03\x05\x03\x06\x03\x08\x07\x08\x08\x08\t\x08\n\x08\x0b\x08\x04\x08\x05\x08\x06\x04\x01\x05\x01\x06\x01\x03\x03\x02\x03\x03\x01\x02\x01\x03\x02\x02\x02\x04\x02\x05\x02"]
Bad pipe message: %s [b'\xaf\x03\x01\x13F=\x85l\\x9c\x19A\x08\xab\x01\xc5\x196\x00\x00\xa6\xc0,\xc00\x00\xa3\x00\x9f\xcc\xa9\xcc']
Bad pipe message: %s [b"3\x0f\x86O\t\x91\xe8\xdeW\xae\xdd\x8aE\x843\xceE\xc5\x00\x00\xf4\xc00\xc0,\xc0(\xc0$\xc0\x14\xc0\n\x00\xa5\x00\xa3\x00\xa1\x00\x9f\x00k\x00j\x00i\x00h\x009\x008\x007\x006\x00\x88\x00\x87\x00\x86\x00\x85\xc0\x19\x00\xa7\x00m\x00:\x00\x89\xc02\xc0.\xc0*\xc0&\xc0\x0f\xc0\x05\x00\x9d\x00=\x005\x00\x84\xc0/\xc0+\xc0'\xc0#\xc0\x13\xc0\t\x00\xa4\x00\xa2\x00\xa0\x00\x9e\x00g\x00@\x00?\x00>\x003\x002\x001\x000\x00\x9a\x00\x99\x00\x98\x00\x97\x00E\x00D\x00C\x00B\xc0\x18\x00\xa6\x00l\x004\x00\x9b\x00F\xc01\xc0", b')\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0\x02\x00\x05\x00\x04\xc0\x12\xc0\x08\x00\x16\x00\x13\x00']
Bad pipe message: %s [b'3']
Bad pipe message: %s [b'\xcb\xbb\x0eXg\xca~+\x92d\xbf\x1b\x16\x8f\x83\x1c\xe7\x9c\x00\x00\xa2\xc0\x14\xc0\n\x009\x008\x007\x006\x00\x88\x00\x87\x00\x86\x00\x85\xc0\x19\x00:\x00\x89\xc0', b'\x05\x005\x00\x84\xc0\x13\xc0\t\x003\x002\x00']
Bad pipe message: %s [b"\xe3\xff\x9b\xb2&h\xfe\xca\xc6\xd5\xb3\xa7\xcc\n\xa4P\t\xe2\x00\x00\x86\xc00\xc0,\xc0(\xc0$\xc0\x14\xc0\n\x00\xa5\x00\xa3\x00\xa1\x00\x9f\x00k\x00j\x00i\x00h\x009\x008\x007\x006\xc02\xc0.\xc0*\xc0&\xc0\x0f\xc0\x05\x00\x9d\x00=\x005\xc0/\xc0+\xc0'\xc0#\xc0\x13\xc0\t\x00\xa4\x00\xa2\x00\xa0\x00\x9e\x00g\x00@\x00?\x00>\x003\x002\x001\x000\xc01\xc0-\xc0)\xc0%\xc0\x0e\xc0\x04\x00\x9c\x00<\x00/\x00\x9a\x00\x99\x00\x98\x00\x97\x00\x96\x00\x07\xc0\x11\xc0\x07\xc0\x0c\xc0\x02\x00\x05\x00\x04\x00\xff\x02\x01\x00\x00g\x00\x00\x00\x0e\x00\x0c\x00\x00\t127.0.0.1\x00\x0b\x00\x04\x03\x00\x01"]
Bad pipe message: %s [b'\x05\xf7#\x11\x0b\xefT\xd0\xbdT\xa6\x89\xd5\xe1', b'\xe6\xfd\x00\x00\xa2\xc0\x14\xc0\n\x009\x008\x007\x006\x00\x88\x00\x87\x00\x86\x00\x85\xc0\x19\x00:\x00\x89\xc0\x0f\xc0\x05\x005\x00\x84\xc0\x13\xc0\t\x003\x002\x001\x000\x00\x9a\x00\x99\x00\x98\x00\x97\x00']
Bad pipe message: %s [b'D\x00C\x00B\xc0\x18\x004\x00\x9b\x00F\xc0\x0e\xc0\x04\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0\x02\x00\x05\x00\x04\xc0\x12\xc0\x08\x00\x16\x00\x13\x00\x10\x00\r\xc0\x17\x00\x1b\xc0\r\xc0\x03\x00\n\x00\x15\x00\x12\x00']
Bad pipe message: %s [b'\n']
Bad pipe message: %s [b'0\x00\x9a\x00\x99\x00\x98\x00\x97\x00E\x00D\x00C\x00B\xc0\x18\x004\x00\x9b\x00F\xc0\x0e\xc0\x04\x00/\x00\x96\x00A\x00\x07\xc0\x11\xc0\x07\xc0\x16\x00\x18\xc0\x0c\xc0']
Bad pipe message: %s [b"\xaa\xc0\xaf\xc0\xad\xc0\xa3\xc0\x9f\xc0]\xc0a\xc0W\xc0S\xc0+\xc0/\x00\xa2\x00\x9e\xc0\xae\xc0\xac\xc0\xa2\xc0\x9e\xc0\\xc0`\xc0V\xc0R\xc0$\xc0(\x00k\x00j\xc0s\xc0w\x00\xc4\x00\xc3\xc0#\xc0'\x00g\x00@\xc0r\xc0v\x00\xbe\x00\xbd\xc0\n\xc0\x14\x009\x008\x00\x88\x00\x87\xc0\t\xc0\x13\x003\x002\x00\x9a\x00\x99\x00E\x00D\xc0\x07\xc0\x11\xc0\x08\xc0\x12\x00\x16\x00\x13\x00\x9d\xc0\xa1\xc0\x9d\xc0Q\x00\x9c\xc0\xa0\xc0\x9c\xc0P\x00=\x00\xc0\x00<\x00\xba\x005\x00\x84\x00/\x00\x96\x00A\x00\x05\x00\n\x00\xff\x01\x00\x00j\x00\x00\x00\x0e\x00\x0c\x00\x00\t1"]
Bad pipe message: %s [b'\r\xc0\x17\x00\x1b\xc0\r\xc0\x03\x00\n\x00\x15\x00\x12']
Bad pipe message: %s [b'\x0c\x00\x1a\x00\t\x00\x14\x00\x11\x00\x19\x00\x08\x00']
Bad pipe message: %s [b'.0.0.1\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x00\x0c\x00\n\x00\x1d\x00\x17\x00\x1e\x00\x19\x00\x18\x00#\x00\x00\x00\x16\x00\x00\x00\x17\x00\x00\x00\r\x000\x00.\x04', b'\x03\x06', b'\x07\x08']
Bad pipe message: %s [b'\x05']
Bad pipe message: %s [b'\x17\x00\x03\xc0\x10']
Bad pipe message: %s [b'\t\x08\n\x08\x0b\x08\x04']
Bad pipe message: %s [b'\x08\x06\x04\x01\x05\x01\x06', b'', b'\x03\x03']
Bad pipe message: %s [b'']
Bad pipe message: %s [b'', b'\x02']
Bad pipe message: %s [b'\x05\x02\x06']
Uploading test_clean_df.csv (< 1 MB): 705MB [00:11, 63.4MB/s]

[Handler.handle()] Failed to read or parse request from socket: Expecting value: line 1 column 1 (char 0)
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xe4 in position 4: invalid continuation byte
[Handler.handle()] Failed to handle request due to exception class=JSONDecodeError, message=Expecting value: line 1 column 1 (char 0)
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xe4 in position 4: invalid continuation byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xe4 in position 4: invalid continuation byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xe4 in position 4: invalid continuation byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xce in position 13: invalid continuation byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xce in position 13: invalid continuation byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xc0 in position 13: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xc0 in position 13: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xaf in position 4: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xaf in position 4: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xa2 in position 12: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xa2 in position 12: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xaf in position 4: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xaf in position 4: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xd0 in position 12: invalid continuation byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xd0 in position 12: invalid continuation byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0x92 in position 14: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0x92 in position 14: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xce in position 4: invalid continuation byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xce in position 4: invalid continuation byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0xb6 in position 12: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8' codec can't decode byte 0xb6 in position 12: invalid start byte
[Handler.handle()] Failed to read or parse request from socket: 'utf-8' codec can't decode byte 0x89 in position 4: invalid start byte
[Handler.handle()] Failed to handle request due to exception class=UnicodeDecodeError, message='utf-8'`

It does not seem to depend on what code is being executed and appears transient at times. Is this impacting any of the data or is there a way to silence this message? I note that in some of the messages besides the garbled messages are strings such as "127.0.0.1" which could be related to the server connection. Thanks for your help in debugging this!

Activity

  1. smurli commented on Aug 31, 2023

    @smurli
    smurli
    on Aug 31, 2023

    I see a similar issue as well with Azure ML Notebooks, especially for long-running cells. Any further details on why this is happening or anyway to work around it?

  2. markkauf commented on Oct 27, 2026

    @markkauf
    markkauf
    on Oct 27, 2026

    These can be caused by a non-Jupyter TCP connection to the kernel's IOPub pipe port, for example a TLS handshake from a security scanner or endpoint agent probing listening ports on the host. The bytes printed after Bad pipe message: %s are the TLS ClientHello (cipher suite values such as \x13\x02\x13\x03\x13\x01, \xc0,\xc00).

    The pipe is the PULL socket IOPubThread._setup_pipe_in binds with bind_to_random_port("tcp://127.0.0.1") (iostream.py). It is the 127.0.0.1 listener of the kernel process that is not listed in the kernel connection file.

    Any message whose first frame is not the pipe UUID is printed by _handle_pipe_msg:

    print("Bad pipe message: %s", msg, file=sys.__stderr__)

    That write to sys.__stderr__ also appears in the cell output.

    To reproduce, run this in a notebook cell (ipykernel 7.4.0, jupyter_client 8.10.0, pyzmq 27.2.0, tornado 6.6, standalone JupyterLab 4.5.1, Python 3.12.4, RHEL 8). It sends a ZMTP 1.0 identity frame and a one-frame message to the kernel's own pipe port:

    import socket, time, ipykernel
from ipykernel.kernelapp import IPKernelApp
port = IPKernelApp.instance().iopub_thread._pipe_port
print(ipykernel.__version__, "pipe port", port)
with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
    s.sendall(b"\x01\x00" + b"\x06\x00hello")
time.sleep(1)

    Cell output:

    7.4.0 pipe port 40855
Bad pipe message: %s [b'hello']

    A TLS handshake to the same port (openssl s_client -connect 127.0.0.1:<pipe_port> <<<'') gives the output seen in this issue, with the ClientHello bytes in place of hello:

    Bad pipe message: %s [b"\xd0Vs\x05\xfc\x80\x95\xa8...\x00H\x13\x02\x13\x03\x13\x01\x13\x04\xc0,\xc00\xcc\xa9\xcc\xa8..."]
Bad pipe message: %s [b'\x05\x04\x03\x04\x03\x03\x00-\x00\x02\x01\x01\x003\x00&\x00$\x00\x1d\x00 oO\xcfm@@...']

    Sending the same payload to all six kernel ports from one cell shows which ones produce output on 7.4.0:

    import json, socket, time, ipykernel
from ipykernel import get_connection_file
from ipykernel.kernelapp import IPKernelApp
cf = json.load(open(get_connection_file()))
ports = {p: cf[p] for p in ("shell_port", "control_port", "stdin_port", "hb_port", "iopub_port")}
ports["pipe_port"] = IPKernelApp.instance().iopub_thread._pipe_port
print(ipykernel.__version__)
for name, port in ports.items():
    print(f"--- {name} {port}", flush=True)
    with socket.create_connection(("127.0.0.1", port), timeout=5) as s:
        s.sendall(b"\x01\x00" + b"\x06\x00hello")
    time.sleep(1)

    Example output:

    7.4.0
--- shell_port 39549
ERROR:tornado.general:Uncaught exception in ZMQStream callback
...
ValueError: DELIM not in msg_list
--- control_port 35701
ERROR:tornado.general:Uncaught exception in ZMQStream callback
...
ValueError: DELIM not in msg_list
--- stdin_port 59875
--- hb_port 55749
--- iopub_port 41793
--- pipe_port 43821
Bad pipe message: %s [b'hello']

    This has the same trigger as #1580 (fixed in #1581), where the same kind of connection to the shell and control ports produced ValueError: DELIM not in msg_list tracebacks in the notebook. The pipe message is a separate code path that predates ipykernel 7 (this issue was opened in April 2023), and with #1581 merged it is the remaining place where these connections show up in cell output. Logging it through self.log (or at debug level) instead of printing to sys.__stderr__ would keep it out of cell output, which was also the goal of #1246.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions