Repository navigation
uwz(kqueue): fix duplicate completion queueing and rearm state (XEV-003, XEV-004) - #1
Merged
Coder-Blue merged 3 commits intoOct 25, 2026
Conversation
When a descriptor is already ready at submission time, submit() returns the event and the callback runs from the completion queue. A callback that returns .rearm left the completion .dead while its kqueue filter stayed registered, so a later cancel was ignored and the next event queued the completion again. Under load this logged "invalid state in submission queue state=.active" for accept and recv completions. A registered completion that rearms is now restored to .active with its stale result cleared. Completions that were never registered keep the resubmit path. Ledger: XEV-003 Upstream: mitchellh#245 (open, unmerged; commit taken from that pull request with its author preserved) Invariant: a completion that is still registered with kqueue and returns .rearm stays .active, so cancellation and re-submission see its true state. Test evidence: "kqueue: cancel after a rearm of a completion reported at submit" fails on the base and passes with this commit; downstream, 12 of 28 64-connection server starts logged the invalid-state error before and 0 of 29 after. Retirement criterion: the fork base contains an equivalent fix.
kevent reports a level-triggered filter again on every call until its callback drains it. submit() issues kevent more than once (several submit calls per tick, and a loop over batches), so a completion that was already waiting in the completion queue was queued a second time. The queue corrupted (the assert in queue.zig push) or perform() ran twice, so a second read or accept consumed data or a connection whose result was then overwritten. Skip an event whose completion is already queued, before perform() runs. Ledger: XEV-004 Upstream: None (downstream; related reports are mitchellh#169 and mitchellh#122 and are not updated) Invariant: a completion is present in the completion queue at most once, and perform() runs once per delivered event. Test evidence: "kqueue: a still-readable completion is queued once across submits" fails without the guard (expected 2 reads, found 1: a byte is lost) and passes with it; downstream the cluster test "programmatic shutdown drains the signal watcher" panics at queue.zig:24 without it. Retirement criterion: upstream or the fork base de-duplicates level-triggered events inside submit().
8 of 16 tasks
XEV-003 and XEV-004 cover the submit-reported rearm and duplicate event paths, but nested submits from callbacks can still requeue the completion currently being dispatched or a completion from the outer tick batch. The no_wait path also dropped staged EV_DELETE changes, and EV_ERROR and non-kqueue rearm paths left the active count unbalanced. Guard both dispatch loops by state, threadpool mode, and completion-queue membership; flush pending deletes before a no_wait tick returns; balance EV_ERROR; and resubmit non-kqueue rearm completions in `.adding` state. Queued-result cancellation remains ordered after the callback because replacing an already-consumed accept or read result can lose a socket or data; that pre-existing limitation is documented in stop_completion. Ledger: XEV-005 Upstream: None (downstream; related to mitchellh#169, mitchellh#122, and mitchellh#224) Invariant: A completion is performed and called back at most once per dispatch, staged deletes reach kqueue before no_wait returns, and every EV_ERROR or non-kqueue rearm keeps the active count balanced. Test evidence: "kqueue: nested submit does not requeue the current completion", "kqueue: nested submit does not double-dispatch an outer batch", "kqueue: EV_ERROR balances the active count", and "kqueue: rearm from a non-kqueue completion resubmits". Retirement criterion: the fork base contains equivalent dispatch and accounting guards.
Coder-Blue
force-pushed
the
fix/kqueue-active-completion-requeue
branch
from
October 25, 2026 18:55
6afc990 to
86d628a
Compare
Coder-Blue
approved these changes
Oct 25, 2026
Coder-Blue
left a comment
Member
There was a problem hiding this comment.
Approved for XEV-003, XEV-004, and the XEV-005 dispatch hardening. Exact head 86d628a passes 65/65 kqueue tests on macOS arm64 in Debug and ReleaseSafe, 142/146 on Linux (4 skipped), across the macOS/FreeBSD/Windows cross-builds, and with the downstream tree at 647/647 on Linux. Merge with a merge commit so the ledger commits remain distinct.
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two
uwz(kqueue)patches that remove a load-induced invalid state and aqueue corruption on the kqueue backend (macOS, BSD). Base:
uwz/zig-0.16at
cbf01bc432a84cd91a99c1cb1031b46416d044c7.Why
Downstream symptoms on macOS with the 2.1.5 server:
error(libxev_kqueue): invalid state in submission queue state=.activewith
op=acceptat 64 concurrent connections.queue.zig:24 assert(v.next == null)panic incluster: programmatic shutdown drains the signal watcher(also at the current pin).
Mechanism for XEV-004, from tracing every completion push and every
keventreturn in
submit(): a level-triggered filter is reported again by eachkeventcall until its callback drains it. A completion already waiting inthe completion queue was queued again, so
perform()ran twice. A secondread or accept consumed data or a connection and then overwrote the result.
Evidence
Server start, one
wrk -t2 -c64 -d2sburst per start, ReleaseSafe,trial counted only if the server logged
server listeningand wrk completedmore than 1000 requests:
cbf01bcFull downstream suite (645 tests): 633 passed, 10 skipped, 2 failed with the
patches. The 2 failures (
config: default slab carries the router region at the end,kernel: validate_umem accepts page-aligned power-of-two regions)also fail at the current pin and are unrelated.
Alternating A/B throughput, ReleaseFast, 64 connections, loaded host:
control 23.0-23.8k req/s, patched 24.4-24.9k req/s.
Fork tests:
zig build test61/61 on macOS. The new testkqueue: a still-readable completion is queued once across submitsfailswithout the guard (
expected 2, found 1: a byte is lost) and passes with it.zig buildalso succeeds for x86_64-macos, x86_64-linux, and x86_64-windows.PR 224 (mitchellh/libxev) was evaluated and rejected: with it the
invalid-state log still appeared in 12 of 30 starts and a downstream timer
test hung. PR 170 only clears
nextbefore re-enqueueing, which would hide thedouble queue.
Review and merge checklist (docs/dependency_forks.md)
kqueue.zigrangedocs/dependency_forks.mduwz/zig-0.16(no force-push, no rewrite)Zig package hash and Nix hash, in a separate uWebZockets change
scripts/check_libxev_fork.shFollow-up hardening (XEV-005)
The first review found three dispatch gaps that are now included in
86d628ac326d69a097ac42c527b04f3389d20d68:and against a completion queued by a nested
submit();EV_DELETEchanges before ano_waittick returns;EV_ERRORand non-kqueue.rearmactive counts.Queued-result cancellation remains ordered after the callback because replacing
an already-consumed accept or read result can lose a socket or data. That
pre-existing limitation is documented in
stop_completion.Ledger: XEV-005. Regression tests are in
src/backend/kqueue.zig.Exact head
86d628ac326d69a097ac42c527b04f3389d20d68passes65/65 tests on macOS arm64 in both Debug and ReleaseSafe, and 142/146 on
Linux (4 skipped).