Linux arm64: a libdispatch worker crashes on a root queue whose head is DISPATCH_OBJECT_LISTLESS while tasks take turns on an actor #11

Summary

On Linux arm64, a libdispatch worker thread dies with a bad pointer dereference while Swift tasks take turns on an actor:

*** Program crashed: Bad pointer dereference at 0xffffffff89abcdff ***

Thread 1 "DispatchWorker" crashed:

0      0x0000ffaf099753f0 _dispatch_worker_thread + 556 in libdispatch.so
1 [ra] 0x0000ffaf097b582c <unknown> in libc.so.6

0xffffffff89abcdef is DISPATCH_OBJECT_LISTLESS, and the address is that plus 0x10, do_next: the worker took the head of a root queue and the head was an item marked as being on no list. x0, x8, x24 and x25 hold the value at the crash.

It needs none of this package's code. A 60-line program that imports only Dispatch — one actor, detached tasks that call it and yield, a DispatchSemaphore the main thread waits on — crashes about once in 130 runs of six seconds each on the ubuntu-24.04-arm runner, under Swift 6.3.3 and 6.4 alike, and not once in 2546 runs on x86_64. In this repository's CI it shows as the Ubuntu 24.04 · Swift 6.x (arm64) jobs failing in the asynchronous locks' measurements, about one run in two.

The toolchain's own swift build has also died inside libdispatch on that runner, once, with a different signature; it is recorded below, and whether it is the same defect is not known.

The cause is a missing store ordering in libdispatch, described under "The cause" below and reported upstream as swiftlang/swift-corelibs-libdispatch#963.

What was observed

Environment: ubuntu-24.04-arm runner (image 20260828.587, Ubuntu 24.04.5 LTS, 4 cores), toolchains from swiftly, swift-6.3.3-RELEASE and swift-6.4-RELEASE, aarch64-unknown-linux-gnu, release builds. The offset is _dispatch_worker_thread + 540 under 6.3.3 and + 556 under 6.4.

First seen on develop in the release XCTest run, after the push that ended at 6eb547f, always in AsyncMutexPerformanceTests or AsyncRWLockPerformanceTests and in a different test each time:

It does not reproduce in a Linux arm64 container on Apple silicon — 2, 4 and 8 cores, the same toolchain binaries, over a hundred runs — so it was narrowed down on the runner itself, from a scratch branch (#10), in three rounds. Each job ran its selection over and over, a process a run, for 35 minutes. Failures over runs:

1. Which commit. The two suites, at six commits, oldest first:

https://github.com/sinoru/swift-synchronization-kit/actions/runs/35518342852 — None of them: for the time it runs, the commit before the push crashes as often as develop, about once in ten to fifteen minutes. What the push changed is exposure. 627215a added some 120 seconds of asynchronous measurements to a job that had about 50, and a crash that had been rare enough to pass fifteen runs began to show in one of two.

2. Whose code. At 2249809, each kind of measurement in a process of its own, four jobs each:

https://github.com/sinoru/swift-synchronization-kit/actions/runs/35520658125 — The actor measurements are the comparison the asynchronous measurements run beside AsyncMutex, the same ones as in #6. On Linux the harness's Atomic is the standard library's, re-exported, so that process runs no lock of this package's at all.

3. Without XCTest or the package. The program below, three jobs for each pair:

https://github.com/sinoru/swift-synchronization-kit/actions/runs/35547187979 — 15 of 2009 on arm64, 0 of 2546 on x86_64, and the same report as in CI.

The reproducer

swift build -c release, then run the product in a loop with SWIFT_BACKTRACE=enable=yes,interactive=no,color=no. Package.swift is a single executable target with no dependencies, in the Swift 5 language mode.

import Dispatch

actor Box {
    private let cycle = (0 ..< 1024).map { ($0 &* 5 &+ 1) % 1024 }
    private var writes = 0

    func step(from index: Int) -> Int {
        writes &+= 1
        return cycle[index]
    }

    var count: Int {
        writes
    }
}

func sample(tasks: Int, iterations: Int) {
    let box = Box()
    let finished = DispatchSemaphore(value: 0)

    for task in 0 ..< tasks {
        Task.detached(priority: .userInitiated) {
            var index = task
            for _ in 0 ..< iterations {
                index = await box.step(from: index)
                await Task.yield()
            }
            finished.signal()
        }
    }
    for _ in 0 ..< tasks {
        finished.wait()
    }

    let checked = DispatchSemaphore(value: 0)
    Task.detached {
        let writes = await box.count
        precondition(writes == tasks * iterations, "the workload did not run")
        checked.signal()
    }
    checked.wait()
}

// The four cases, ten samples of each, as the measurements take them.
for (tasks, iterations) in [(64, 2_000), (8, 10_000), (1, 100_000), (512, 250)] {
    for _ in 0 ..< 10 {
        sample(tasks: tasks, iterations: iterations)
    }
}

The loop that ran it, from the scratch branch's workflow:

runs=0 failures=0
while [ $SECONDS -lt 2100 ]; do
  runs=$((runs + 1))
  if ! "$reproducer" > run.log 2>&1; then
    failures=$((failures + 1))
    grep -A 6 -E "Program crashed|crashed:|Fatal error" run.log
  fi
done
echo "$failures of $runs runs failed"

The cause

When a worker takes the executor's concurrent queue off a root queue, _dispatch_queue_class_invoke writes DISPATCH_OBJECT_LISTLESS into the queue's do_next with a plain store. If the queue cannot run because its width is used up — and a wakeup enqueues it without looking at width — _dispatch_queue_drain_try_lock then clears ENQUEUED with an acquire-only compare-and-swap (casa). Nothing orders the store before the swap. Another thread can see ENQUEUED clear, enqueue the queue again — NULL into do_next, and linked — and have the late LISTLESS land on top. The next worker takes it for the next item and stores it as the head, and the one after dies on it. On x86_64 the locked swap drains the store buffer first, which is why it never crashed there.

Tested on the runner in #14, in two runs (36094940006, 36099069116): libdispatch at swift-6.4.0-RELEASE rebuilt with the toolchain's clang three ways, each loaded in place of the toolchain's under the reproducer.

A litmus test of the pattern alone (StoreOrder/litmus.c in #14) sees the store land after the swap about 2 × 10⁻⁶ of the time on the runner's Neoverse N2 and about 2 × 10⁻⁸ on an Apple M4 Pro, only when the two words are in different cache lines, and never with release on the swap or a store barrier before it. The M4 Pro making it about a hundred times less often is why it would not reproduce locally.

What it does and does not look like

• Not a regression in this package, and not this package: rounds 1 and 2.
• arm64 only, so far. x86_64 ran more often in the same time and never crashed, which fits a missing barrier or an unordered pair of stores somewhere between the concurrency runtime and libdispatch's queues, and does not prove one. The cause above is that barrier.
• On Linux the runtime's global executor does not use libdispatch's global queues directly: for each priority it makes a concurrent queue as wide as the machine has cores, four here, and enqueues jobs on that (getGlobalQueue in DispatchGlobalExecutor.cpp). When the width is used up the queue object itself goes onto a root queue, and libdispatch writes DISPATCH_OBJECT_LISTLESS into an object's do_next when it invokes one (_dispatch_queue_class_invoke) as well as when it creates one. So that value being written to objects that pass through root queues is routine on this path; what is wrong is that one was still, or again, linked as a root queue's head when it happened. The cause above is how: again, with the store arriving late.
• AsyncSemaphorePerformanceTests has never crashed, in CI or in 8500 seconds of round 2, where the other two selections crashed 14 and 8 times in as long. Its tasks yield as the others do. What it lacks is not known; the reproducer's actor and AsyncMutex both hand a job straight from one task to the next.
• Windows arm64: an actor counter ends one increment short in the measurements' comparison #6 is the same workload on the other arm64 that is not Apple's, Windows, where the actor's count ends one short instead. Nothing connects the two yet but that.

A second signature, in the toolchain itself

The Ubuntu 24.04 · Swift 6.4 (arm64, ASan) job failed at 49289a9 in swift build, before any source of this package was compiled: https://github.com/sinoru/swift-synchronization-kit/actions/runs/35632781294/job/106442767580

Building for production...
[Pre-planning 1 / 475]

*** Program crashed: Bad pointer dereference at 0xffff547093b0ab97 ***

Platform: arm64 Linux (Ubuntu 24.04.5 LTS)

Thread 2 "DispatchWorker" crashed:

0      0x0000ff927e01cd08 _dispatch_event_loop_drain + 1228 in libdispatch.so
1 [ra] 0x0000ff927e012084 _dispatch_mgr_invoke + 123 in libdispatch.so
2 [ra] 0x0000ff927e011ff8 _dispatch_mgr_thread + 131 in libdispatch.so
3 [ra] 0x0000ff927e015448 _dispatch_worker_thread + 643 in libdispatch.so

The process is the toolchain's swift-package, planning the build; the other threads were discovering the linker (discoveredLinkerToolsInfo) and constructing settings. --sanitize address instruments what the build would have produced, not the build system, so the row it happened in is incidental. It is the same runner and the same libdispatch.so as in the runs above (build ID 95715b25…, swift-6.4-RELEASE from swiftly), and the first failure of that step in the last 25 runs of the workflow.

It is not the signature above. This is libdispatch's manager thread draining epoll events, not a worker taking the head of a root queue, and the value is not DISPATCH_OBJECT_LISTLESS:

• The address is x8 + 8, and x8 is 0xffff547093b0ab8f, whose complement, 0x0000ab8f6c4f5470, is an address in that process's heap (x20 is 0x0000ab8f6c2aab08).
• The instruction is ldr w9, [x8, #8] followed by a comparison with w28, which holds 0x7fffffff: a reference count read at offset 8 and tested against _OS_OBJECT_GLOBAL_REFCNT, which is what _dispatch_retain_2 begins with.
• In event_epoll.c the drain merges an event into each unote on the muxnote (_dispatch_event_merge_fd, _dispatch_event_merge_signal), and retains the unote's owner first: _dispatch_retain_unote_owner(du), which is _dispatch_retain_2(_dispatch_wref2ptr(du->du_owner_wref)). A weak reference is stored complemented, and _dispatch_wref2ptr is ~.

So the slot read as du_owner_wref held a plain heap pointer rather than a complemented one, which fits a unote that had been freed and its memory reused while its muxnote could still deliver an event. That is a reading of the registers and the sources, not something observed, and one occurrence.

What it adds: a crash inside libdispatch on this runner in a process that runs none of this package's code and was not written for this issue. The cause above does not explain it: this path takes no queue off a root queue. swiftlang/swift-corelibs-libdispatch#949 reports a use-after-free of epoll muxnotes on aarch64 Linux under concurrent FileHandle.readabilityHandler teardown, which may be this one.

Consequences for this package

• The Linux arm64 pairs in swift.yml, under both Swift minors, are marked experimental (continue-on-error), which also means a real failure of the package on Linux arm64 would hide behind this one. The sanitized arm64 rows are not marked: they skip the measurements, and have not crashed in the tests. One has failed in swift build with the second signature, once in 25 runs, and a rerun clears that.
• Nothing in the library is implicated. What crashes is the platform under a workload the measurements happen to produce, and a client's program that produces it would crash the same way. The experimental marks stay until a toolchain carries a fix.

To do

• Search swiftlang/swift-corelibs-libdispatch and swiftlang/swift for the signature before filing. A search for 89abcdef found nothing on 2026-09-20, nor one for LISTLESS, do_next or _dispatch_worker_thread on 2026-09-25.
• Search both for the second signature as well (_dispatch_event_loop_drain, _dispatch_retain_unote_owner, a unote freed under the epoll muxnote), and decide whether it belongs in the same report or one of its own. It does not belong in aarch64: LISTLESS store in queue invoke can land after the queue is re-enqueued swiftlang/swift-corelibs-libdispatch#963; Use-after-free in epoll muxnote bookkeeping under concurrent FileHandle.readabilityHandler teardown (Linux) swiftlang/swift-corelibs-libdispatch#949 may be it.
• Shrink the reproducer: which of the four cases it takes, whether the priority or the Task.yield() matters, whether a plain dispatch_async workload on a width-limited concurrent queue does it without Swift concurrency at all — which would say which repository it belongs to. Not needed: the barrier experiment put it in libdispatch.
• Try it on arm64 hardware other than the hosted runner's. The litmus test on an Apple M4 Pro: the reordering happens there, a hundred times less often.
• File upstream: aarch64: LISTLESS store in queue invoke can land after the queue is re-enqueued swiftlang/swift-corelibs-libdispatch#963.
• Find what spares AsyncSemaphorePerformanceTests.
• Note further occurrences of the second signature here: one is not a rate.
• Take experimental off the Linux arm64 pairs in swift.yml once a toolchain carries a fix for aarch64: LISTLESS store in queue invoke can land after the queue is re-enqueued swiftlang/swift-corelibs-libdispatch#963.