[endel]

On kqueue, when submit() registers more than 256 changes at once and some are already ready, a completion can be queued twice, and its callback run a second time after the first run freed it. tick() can then also write past its 256-entry change list, onto the stack.

Queued twice. submit() passes an event list with each batch of 256 changes. Read and write filters are added level-triggered, so each batch's kevent reports again every ready completion an earlier batch registered, and submit() pushes it onto the intrusive completion queue again. A queued completion also stays registered, so the wait in tick() can report it and run its callback, and then the queue runs it again. When the first callback frees the completion, as a server does with a connection that reads EOF, popping the queue reads freed memory.

Past the change list. tick() writes an EV_DELETE for each completion it disarms from the queue, and checks whether the list is full only after the write; the assert(changes <= events.len) is compiled out in ReleaseFast. When the wait's own disarms have already filled the list, the next write lands past the array.

This registers with EV_RECEIPT and an event list exactly as long as the change list, so submit() gets one receipt per change and no pending events. A change that failed is queued with its error, as before; everything else is reported once, by the wait in the same tick(). Nothing in the completion queue is then registered with kqueue, so tick() no longer deletes anything from it, and the write past the list goes with it. A completion whose registration failed and that asks to rearm is registered again, rather than left counted active and never reported. This also removes the NOTE in submit() about memory corruption from processing completions there, which may well have been this.

We hit this in a game server on quic-zig on macOS: about 1,000 WebSocket clients disconnecting together crashed a ReleaseFast build within 3 to 5 rounds. Guard Malloc caught the use-after-free in queue.pop under Loop.run, and a hardware watchpoint caught events[changes] = ev in tick() overwriting the server's state on the stack. With this change it ran 8 rounds of 1,024 clients, and 8 of 512 under Guard Malloc.

Stacked

Only the last commit is this change. The ones before it are #245 and #224 (with a zig fmt of #224's tests), which change the same code in submit() and tick(). I'll rebase this once they're merged or closed.

Test

"kqueue: more ready reads than one submit batch each complete once": 600 pipes, each with a byte written and its write end closed, so every read is ready before it is submitted. Each callback must run once, and the loop must end with nothing active. The test raises the open-file soft limit to 1,200 if it is lower, and is skipped if the hard limit is below that. Without the change, a callback runs a second time and its read returns EOF.

The branch sits on #224, from before main moved to Zig 0.17, so it was tested with Zig 0.16.0:

• macOS (aarch64, kqueue): zig build test, 72 passed
• Linux (aarch64, epoll and io_uring): 146 passed, 3 skipped
• zig build -Demit-example -Demit-bench, and zig build for every target in the CI matrix

And on current main with Zig 0.17.0, together with our other open PRs: macOS 75 passed, 1 skipped; Linux 155 passed, 5 skipped; the examples, benchmarks and every CI target build.

CI on main fails until #250 moves the workflow to Zig 0.17, so this PR's checks will fail the same way until then.

🤖 Generated with Claude Code