Skip to content

uwz(kqueue): fix duplicate completion queueing and rearm state (XEV-003, XEV-004) - #1

Merged
Coder-Blue merged 3 commits into
farbenbuilds:uwz/zig-0.16from
crazyads69:fix/kqueue-active-completion-requeue
Oct 25, 2026
Merged

Coder-Blue merged 3 commits into
farbenbuilds:uwz/zig-0.16from
crazyads69:fix/kqueue-active-completion-requeue

Conversation

@crazyads69

@crazyads69 crazyads69 commented Oct 25, 2026 •

Copy link
Copy Markdown

Summary

Two uwz(kqueue) patches that remove a load-induced invalid state and a
queue corruption on the kqueue backend (macOS, BSD). Base: uwz/zig-0.16
at cbf01bc432a84cd91a99c1cb1031b46416d044c7.

Ledger Commit Upstream reference
XEV-003 keep a rearmed completion active after submit reported it mitchellh#245 (open, unmerged; author preserved). Reference only: nothing is sent upstream.
XEV-004 queue a still-readable completion once None (downstream). Related reports: mitchellh#169 and mitchellh#122.

Why

Downstream symptoms on macOS with the 2.1.5 server:

  • error(libxev_kqueue): invalid state in submission queue state=.active
    with op=accept at 64 concurrent connections.
  • queue.zig:24 assert(v.next == null) panic in
    cluster: programmatic shutdown drains the signal watcher
    (also at the current pin).

Mechanism for XEV-004, from tracing every completion push and every kevent
return in submit(): a level-triggered filter is reported again by each
kevent call until its callback drains it. A completion already waiting in
the completion queue was queued again, so perform() ran twice. A second
read or accept consumed data or a connection and then overwrote the result.

Evidence

Server start, one wrk -t2 -c64 -d2s burst per start, ReleaseSafe,
trial counted only if the server logged server listening and wrk completed
more than 1000 requests:

Libxev Valid trials Trials with the invalid-state log
pin cbf01bc 28 12
with these two commits 29 0

Full downstream suite (645 tests): 633 passed, 10 skipped, 2 failed with the
patches. The 2 failures (config: default slab carries the router region at the end, kernel: validate_umem accepts page-aligned power-of-two regions)
also fail at the current pin and are unrelated.

Alternating A/B throughput, ReleaseFast, 64 connections, loaded host:
control 23.0-23.8k req/s, patched 24.4-24.9k req/s.

Fork tests: zig build test 61/61 on macOS. The new test
kqueue: a still-readable completion is queued once across submits fails
without the guard (expected 2, found 1: a byte is lost) and passes with it.
zig build also succeeds for x86_64-macos, x86_64-linux, and x86_64-windows.

PR 224 (mitchellh/libxev) was evaluated and rejected: with it the
invalid-state log still appeared in 12 of 30 starts and a downstream timer
test hung. PR 170 only clears next before re-enqueueing, which would hide the
double queue.

Review and merge checklist (docs/dependency_forks.md)

  • Build/vendor and transport/runtime review of the kqueue.zig range
  • Add ledger rows XEV-003 and XEV-004 in uWebZockets docs/dependency_forks.md
  • Merge into uwz/zig-0.16 (no force-push, no rewrite)
  • Pin the resulting full 40-hex commit in the four zon views with the
    Zig package hash and Nix hash, in a separate uWebZockets change
  • Re-run scripts/check_libxev_fork.sh

Follow-up hardening (XEV-005)

The first review found three dispatch gaps that are now included in
86d628ac326d69a097ac42c527b04f3389d20d68:

  • guard both dispatch loops against a completion currently being called back
    and against a completion queued by a nested submit();
  • flush staged EV_DELETE changes before a no_wait tick returns;
  • balance EV_ERROR and non-kqueue .rearm active counts.

Queued-result cancellation remains ordered after the callback because replacing
an already-consumed accept or read result can lose a socket or data. That
pre-existing limitation is documented in stop_completion.

Ledger: XEV-005. Regression tests are in src/backend/kqueue.zig.
Exact head 86d628ac326d69a097ac42c527b04f3389d20d68 passes
65/65 tests on macOS arm64 in both Debug and ReleaseSafe, and 142/146 on
Linux (4 skipped).

endel and others added 2 commits October 25, 2026 21:46
When a descriptor is already ready at submission time, submit() returns the
event and the callback runs from the completion queue. A callback that
returns .rearm left the completion .dead while its kqueue filter stayed
registered, so a later cancel was ignored and the next event queued the
completion again. Under load this logged "invalid state in submission queue
state=.active" for accept and recv completions.

A registered completion that rearms is now restored to .active with its stale
result cleared. Completions that were never registered keep the resubmit path.

Ledger: XEV-003
Upstream: mitchellh#245 (open, unmerged;
  commit taken from that pull request with its author preserved)
Invariant: a completion that is still registered with kqueue and returns
  .rearm stays .active, so cancellation and re-submission see its true state.
Test evidence: "kqueue: cancel after a rearm of a completion reported at
  submit" fails on the base and passes with this commit; downstream, 12 of 28
  64-connection server starts logged the invalid-state error before and 0 of
  29 after.
Retirement criterion: the fork base contains an equivalent fix.
kevent reports a level-triggered filter again on every call until its
callback drains it. submit() issues kevent more than once (several submit
calls per tick, and a loop over batches), so a completion that was already
waiting in the completion queue was queued a second time. The queue corrupted
(the assert in queue.zig push) or perform() ran twice, so a second read or
accept consumed data or a connection whose result was then overwritten.

Skip an event whose completion is already queued, before perform() runs.

Ledger: XEV-004
Upstream: None (downstream; related reports are
  mitchellh#169 and
  mitchellh#122 and are not updated)
Invariant: a completion is present in the completion queue at most once, and
  perform() runs once per delivered event.
Test evidence: "kqueue: a still-readable completion is queued once across
  submits" fails without the guard (expected 2 reads, found 1: a byte is
  lost) and passes with it; downstream the cluster test "programmatic
  shutdown drains the signal watcher" panics at queue.zig:24 without it.
Retirement criterion: upstream or the fork base de-duplicates level-triggered
  events inside submit().
XEV-003 and XEV-004 cover the submit-reported rearm and duplicate event
paths, but nested submits from callbacks can still requeue the completion
currently being dispatched or a completion from the outer tick batch. The
no_wait path also dropped staged EV_DELETE changes, and EV_ERROR and
non-kqueue rearm paths left the active count unbalanced.

Guard both dispatch loops by state, threadpool mode, and completion-queue
membership; flush pending deletes before a no_wait tick returns; balance
EV_ERROR; and resubmit non-kqueue rearm completions in `.adding` state.
Queued-result cancellation remains ordered after the callback because
replacing an already-consumed accept or read result can lose a socket or
data; that pre-existing limitation is documented in stop_completion.

Ledger: XEV-005
Upstream: None (downstream; related to mitchellh#169, mitchellh#122, and mitchellh#224)
Invariant: A completion is performed and called back at most once per
  dispatch, staged deletes reach kqueue before no_wait returns, and every
  EV_ERROR or non-kqueue rearm keeps the active count balanced.
Test evidence: "kqueue: nested submit does not requeue the current
  completion", "kqueue: nested submit does not double-dispatch an outer
  batch", "kqueue: EV_ERROR balances the active count", and "kqueue:
  rearm from a non-kqueue completion resubmits".
Retirement criterion: the fork base contains equivalent dispatch and
  accounting guards.
@Coder-Blue
Coder-Blue force-pushed the fix/kqueue-active-completion-requeue branch from 6afc990 to 86d628a Compare October 25, 2026 18:55

@Coder-Blue Coder-Blue left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for XEV-003, XEV-004, and the XEV-005 dispatch hardening. Exact head 86d628a passes 65/65 kqueue tests on macOS arm64 in Debug and ReleaseSafe, 142/146 on Linux (4 skipped), across the macOS/FreeBSD/Windows cross-builds, and with the downstream tree at 647/647 on Linux. Merge with a merge commit so the ledger commits remain distinct.

@Coder-Blue
Coder-Blue merged commit c49f6df into farbenbuilds:uwz/zig-0.16 Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants