Skip to content

Pyjo.Server.Daemon: the data of a connection that the daemon has removed, whose stream reads on while it closes gracefully, is discarded - #554

Merged
dex4er merged 2 commits into
mainfrom
daemon-read-removed
Nov 5, 2026
Merged

dex4er merged 2 commits into
mainfrom
daemon-read-removed

Conversation

@dex4er

@dex4er dex4er commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Pyjo.Reactor.Poll.Pyjo_Reactor_Poll: I/O watcher failed: '<id>' at Pyjo/Server/Daemon.py line 918 on the standard error of t/Pyjo/App_ASGI.py, which passed, in prove (python pypy3.12, Poll) of #553 (job): _read of the daemon got the data of a connection that it had removed already, and self._connections[id] raised KeyError.

Why

The daemon removes the connection of a WebSocket in _finish, which the callback of the event drain of its last write calls. remove of the event loop closes the stream gracefully, and the stream was still writing then, with an empty buffer, as the callback of another write, which came during the emit of drain, waited for the next one. So the stream read on until its next writable event, while the daemon had dropped the connection at once, and a pong of the client that came first was read for a connection that was gone. Mojo makes the connection again there, as Perl autovivifies it, and builds a transaction of the data; the other handlers of the daemon, _write, _finish and _trace, take a connection that is gone already.

Reproduced with t/Pyjo/App_ASGI.py on PyPy 3.12 and Pyjo.Reactor.Poll in a container of Linux on arm64, once in 40 to 120 runs, with the stacks of the removal and of the read recorded.

Changes

  • Pyjo.Server.Daemon: _read discards the data of a connection that the daemon has removed, with an # Unlike Mojolicious: marker.
  • t/stdlib/Server_Daemon.py: Data of a connection that the daemon has removed, a response with Connection: close whose stream is kept writing, as by another callback of drain, until the client has sent data after the response; it fails without the change with the error of the reactor of CI.

Validation

  • The new subtest fails without the change and passes with it, on the asyncio reactor and on Pyjo.Reactor.Poll
  • The whole suite with PYTHONDEVMODE=1 on the asyncio reactor and on Pyjo.Reactor.Poll
  • 120 runs of t/Pyjo/App_ASGI.py on PyPy 3.12 with Pyjo.Reactor.Poll in the container, without the error, where it came once in 40 and once in 120 before
  • trunk check
  • CI

🤖 Generated with Claude Code

dex4er and others added 2 commits November 5, 2026 14:59
…ay be due a millisecond early

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ved, whose stream reads on while it closes gracefully, is discarded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Tests

Job Tests Passed ✅ Skipped ⚠️ Failed ❌ Log
prove (python 3.12, Asyncio) 7369 7341 28 0 ✅ 127300339559
prove (python 3.12, Poll) 7341 7311 30 0 ✅ 127300339410
prove (python 3.13, Asyncio) 7369 7341 28 0 ✅ 127300339336
prove (python 3.13, Poll) 7341 7311 30 0 ✅ 127300339233
prove (python 3.14, Asyncio) 7400 7377 23 0 ✅ 127300339160
prove (python 3.14, Poll) 7372 7347 25 0 ✅ 127300339275
prove (python 3.15, Asyncio) 7400 7377 23 0 ✅ 127300339322
prove (python 3.15, Poll) 7372 7347 25 0 ✅ 127300338642
prove (python 3.15t, Asyncio) 7400 7377 23 0 ✅ 127300339369
prove (python 3.15t, Poll) 7372 7347 25 0 ✅ 127300339381
prove (python mise, Asyncio) 7479 7445 34 0 ✅ 127300338630
prove (python mise, Asyncio, Windows) 7206 7142 64 0 ✅ 127300338938
prove (python mise, Poll) 7451 7415 36 0 ✅ 127300338957
prove (python mise, Poll, Windows) 7178 7112 66 0 ✅ 127300339137
prove (python pypy3.12, Asyncio) 7369 7341 28 0 ✅ 127300339489
prove (python pypy3.12, Poll) 7341 7311 30 0 ✅ 127300339463
pytest (python 3.12, ubuntu-26.04-arm) 434 409 25 0 ✅ 127300338671
pytest (python 3.15, ubuntu-26.04-arm) 434 414 20 0 ✅ 127300338833
pytest (python 3.15, windows-2025) 434 387 47 0 ✅ 127300338688
servers (python 3.12) 45 45 0 0 ✅ 127300338307
servers (python 3.15) 44 44 0 0 ✅ 127300338555
All 119151 118541 610 0 ✅

The run of the workflow

@dex4er
dex4er merged commit 7b09754 into main Nov 5, 2026
37 checks passed
@dex4er
dex4er deleted the daemon-read-removed branch November 5, 2026 14:24
dex4er added a commit to pyjoyment/Pyjo-Reactor-Winloop that referenced this pull request Nov 5, 2026
Pyjoyment of Git main with pyjoyment/Pyjoyment#553, a sleep of
Pyjo.Steps in its tests on an event loop of libuv, whose timer may be
due a millisecond early, and pyjoyment/Pyjoyment#554, the data of a
connection that the daemon has removed discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dex4er added a commit to pyjoyment/Pyjo-Reactor-Winloop that referenced this pull request Nov 5, 2026
…ime, and a timer not due before its time (#6)

* Pyjo.Reactor.Winloop: a socket that loops watch, polled by one at a time

libuv of Windows polls a socket with an exclusive request of AFD, which
ends the request of any other loop for the same socket, also of a copy
of it, and that loop polls it again, in a busy loop of C with no event
at all. The workers of Pyjo.Server.Prethread watch the listen sockets of
the manager on Windows, the same sockets in every thread: four idle
loops took 240% of the processors and starved the process and the tests
next to it, with or without the GIL.

_Loop keeps the readers and the writers of each socket, and a registry
of the process has the loops that watch it, the owner that libuv polls
it for, and the loops that run and wait for it. The owner hands it over
to the next one after each of its events, so that the connections go to
every worker in turn, and at the end of its run; a loop changes the
polls of libuv only in its thread. A socket that one loop watches goes
to libuv as before.

Closes #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* uv.lock: Pyjoyment of Git main at 1fc984a

Pyjoyment of Git main with pyjoyment/Pyjoyment#550 and
pyjoyment/Pyjoyment#552, the tests of a page of GitHub without its title
in the tutorial and of the servers of Hot and Reload that did not answer
or did not stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* uv.lock: Pyjoyment of Git main at 7b09754

Pyjoyment of Git main with pyjoyment/Pyjoyment#553, a sleep of
Pyjo.Steps in its tests on an event loop of libuv, whose timer may be
due a millisecond early, and pyjoyment/Pyjoyment#554, the data of a
connection that the daemon has removed discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pyjo.Reactor.Winloop: a timer not due before its time

libuv counts a timer in whole milliseconds from uv_now, its clock
rounded down, and Winloop rounds the delay to them, as uvloop does, so a
timer, as asyncio.sleep, was due up to a millisecond before its time,
where the loop of asyncio is never early: a sleep of 2 ms took 1.5 ms.
call_later of _Loop, which call_at of Winloop calls too, rounds the
delay up to a millisecond and adds the one that the clock loses, as
_Loop of Pyjo.Reactor.Uvloop does, after MagicStack/uvloop#755.

The subtest of Pyjo-Reactor-Uvloop measures with time.perf_counter,
which time.monotonic is on Windows since Python 3.13, and which counted
in the ticks of the system of 15.6 ms before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@dex4er dex4er added this to the 0.3.0 milestone Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant