Skip to content

Measure through a benchmark package of its own #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sinoru merged 3 commits into from
Oct 11, 2026
Merged

Measure through a benchmark package of its own #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
232b9cd
Measure through a benchmark package of its own
sinoru Oct 10, 2026
1e8e6ff
Take the measurements out of the test suites
sinoru Oct 9, 2026
2903175
Point the README at the benchmark package and its numbers
sinoru Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/apple-platforms.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
# names of the suites' types; the steps below say how it is used here.
env:
DEVELOPER_DIR: ${{ matrix.xcode.developer_dir }}
SUITES: PerformanceTests|StressTests
SUITES: StressTests
strategy:
fail-fast: false
# Two axes, crossed. The Xcode axis is the one swift.yml already runs: the
Expand Down Expand Up @@ -267,11 +267,11 @@ jobs:
#
# Both configurations in turn, as swift.yml does and for its reasons:
# release is the library as a client gets it, where the optimizer's bugs
# and the measurements are, and it is built without testability so that
# what is stressed and measured is what ships; debug is where `@testable
# import` works unasked, and where the unit tests that need it are
# compiled at all. Release matters most on the watchOS device row, whose
# 32-bit pointer branch is compiled nowhere else. Both name the same
# are, and it is built without testability so that what is stressed is
# what ships; debug is where `@testable import` works unasked, and where
# the unit tests that need it are compiled at all. Release matters most
# on the watchOS device row, whose 32-bit pointer branch is compiled
# nowhere else. Both name the same
# derived data, where Xcode keeps each configuration's products apart,
# and the run that follows reuses what was built here instead of
# compiling the package again.
Expand Down
153 changes: 88 additions & 65 deletions .github/workflows/swift.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,16 @@ name: Swift
# would only run the same commit twice.
#
# And only for a change that can alter what is built here: the sources, the
# tests, the manifest and what it resolved to, and this file. A commit to the
# README or the changelog runs nothing. A branch filter and a path filter must
# both be satisfied.
# tests, the benchmarks, the manifest and what it resolved to, and this file.
# A commit to the README or the changelog runs nothing. A branch filter and a
# path filter must both be satisfied.
on:
push:
branches: [main, develop, "release/**", "hotfix/**"]
paths: &paths
- Sources/**
- Tests/**
- Benchmarks/**
- Package.swift
- Package.resolved
- .github/workflows/swift.yml
Expand Down Expand Up @@ -60,8 +61,9 @@ jobs:
# A dropped wake in a lock does not fail a test, it stops one, and without
# this the only backstop is GitHub's 360-minute default — six hours this
# repository has already paid once. The joins inside the suite are bounded
# by hand for the same reason; this covers whatever they cannot.
timeout-minutes: 30
# by hand for the same reason; this covers whatever they cannot, with room
# for the benchmarks on the rows that run them.
timeout-minutes: 45
# All are inert where the matrix leaves them unset. DEVELOPER_DIR selects
# the toolchain on macOS by pointing at a pinned Xcode: the runner keeps a
# minor-level symlink at the current patch (Xcode_26.6.app ->
Expand All @@ -86,7 +88,7 @@ jobs:
TEST_OPTIONS: ${{ matrix.test_options }}
SHORT_TESTS: ${{ matrix.short_tests }}
ANDROID: ${{ matrix.android }}
SUITES: PerformanceTests|StressTests
SUITES: StressTests
strategy:
fail-fast: false
# Two axes, crossed: every target under every Swift minor, less the pairs
Expand Down Expand Up @@ -150,9 +152,12 @@ jobs:
macos: macOS 27
developer_dir: /Applications/Xcode_27.0.app/Contents/Developer

# macOS has no entry of its own: its runner and its Xcode are its
# minor's. The Intel image is the one place the suite runs on x86_64
# Darwin, a slice that is otherwise compiled and never run.
# macOS has an entry only to say it runs the benchmarks: its runner
# and its Xcode are its minor's. The Intel image is the one place
# the suite runs on x86_64 Darwin, a slice that is otherwise
# compiled and never run.
- target: macOS
benchmarks: true
- target: macOS Intel
os: macOS 26
variant: Intel
Expand All @@ -168,6 +173,7 @@ jobs:
- target: Linux
os: Ubuntu 24.04
runner: ubuntu-24.04
benchmarks: true
- target: Linux arm64
os: Ubuntu 24.04
variant: arm64
Expand Down Expand Up @@ -217,22 +223,19 @@ jobs:
# and the single-threaded parts of the rest. The threaded flavor is
# in development snapshots only, and is not built here.
#
# Two departures from the other targets. XCTest is off: with the
# measurement suites compiled out there is no XCTest case left, and
# the corelibs runner traps on an empty list rather than reporting
# none. And the stress suites run at their short scale: the Wasm
# SDK emits no tail calls, so an `await` that never actually
# suspends nests a frame instead of unwinding one, and the long
# scale's thousands of them in a row exhaust the interpreter's call
# stack, which `swift test` has no way to raise.
# One departure from the other targets: the stress suites run at
# their short scale. The Wasm SDK emits no tail calls, so an
# `await` that never actually suspends nests a frame instead of
# unwinding one, and the long scale's thousands of them in a row
# exhaust the interpreter's call stack, which `swift test` has no
# way to raise.
- target: WebAssembly wasip1
os: WebAssembly
variant: wasip1
runner: ubuntu-24.04
pinned: true
swift_sdk: swift-{0}-RELEASE_wasm
sdk_url: https://download.swift.org/swift-{0}-release/wasm-sdk/swift-{0}-RELEASE/swift-{0}-RELEASE_wasm.artifactbundle.tar.gz
test_options: --disable-xctest
short_tests: true
# Cross-compiles to Android through the Swift SDK for it, and runs the
# tests on an emulator. This is the only target where the bionic side
Expand Down Expand Up @@ -370,7 +373,9 @@ jobs:
# Windows on arm64 loses an increment inside an `actor`, the
# measurements' comparison and none of this package's code: 127999
# of 128000, always short by one. Under 6.3 every time so far, under
# 6.4 one time in four. (#6)
# 6.4 one time in four. The measurements have since moved to the
# benchmark package, which does not run on Windows, so the pair is
# kept marked until a run without them shows it clear. (#6)
- { swift: "6.3", target: Windows arm64, experimental: true }
- { swift: "6.4", target: Windows arm64, experimental: true }
# The 6.4 toolchain's WasmKit implements `poll_oneoff`, so a sleep
Expand All @@ -387,8 +392,10 @@ jobs:
# asynchronous measurements: about one run in two. A program of an
# actor, detached tasks and `Task.yield`, with none of this
# package in it, does the same on this runner once in 130 runs and
# on x86_64 never. The sanitized arm64 rows skip the measurements
# and are not marked. (#11)
# on x86_64 never. The measurements have since moved to the
# benchmark package, which this row does not run, so the pair is
# kept marked until a run without them shows it clear; the
# sanitized arm64 rows never ran them and are not marked. (#11)
- { swift: "6.3", target: Linux arm64, experimental: true }
- { swift: "6.4", target: Linux arm64, experimental: true }

Expand Down Expand Up @@ -525,11 +532,10 @@ jobs:
# Release is the library as a client gets it, and a lock is a type whose
# bugs the optimizer is entitled to create: reordering, dead stores,
# accesses folded into a register. A debug run cannot see any of them, so
# the stress suites run in release, and the measurements with them, which
# say nothing unoptimized. Neither needs more than the public interface,
# and so nothing in release is compiled with `-enable-testing`, which
# publishes internal symbols and costs the optimizer some of what it may
# assume: what is stressed and measured is what ships.
# the stress suites run in release. They need no more than the public
# interface, and so nothing in release is compiled with
# `-enable-testing`, which publishes internal symbols and costs the
# optimizer some of what it may assume: what is stressed is what ships.
#
# Debug is where `@testable import` works without being asked for, so the
# unit tests run there. The files that reach internal declarations are
Expand Down Expand Up @@ -567,21 +573,24 @@ jobs:
# can only run where a full toolchain exists.
#
# The stress suites run at length on every push, on every row but the
# ones that say `short_tests`: the flag turns their
# repetition up fiftyfold, a tenth of that under the sanitizer, and
# what that costs — some ten seconds on a runner — is less than the
# measurements take, while what it finds, a wake lost in one
# interleaving out of many, is what a change to a wait or wake path
# most needs found before it merges. `stressScale` in the test utils
# says how the flag is read, and why the factor is not higher. Only the
# release build takes it, being the only one that runs them.
# ones that say `short_tests`: the flag turns their repetition up
# fiftyfold, a tenth of that under the sanitizer, and what that costs —
# some ten seconds on a runner — is small beside what it finds, a wake
# lost in one interleaving out of many, which is what a change to a wait
# or wake path most needs found before it merges. `stressScale` in the
# test utils says how the flag is read, and why the factor is not
# higher. Only the release build takes it, being the only one that runs
# them.
#
# XCTest is left out of both builds: no XCTest case is left in the
# package, and the step that runs the tests says why that matters.
- name: swift build --build-tests
if: ${{ !matrix.skip_tests }}
shell: bash
run: |
for configuration in debug release; do
echo "::group::$configuration"
test_flags="-c $configuration${SWIFT_SDK:+ --swift-sdk $SWIFT_SDK}${SANITIZE:+ --sanitize $SANITIZE} $TEST_OPTIONS"
test_flags="-c $configuration --disable-xctest${SWIFT_SDK:+ --swift-sdk $SWIFT_SDK}${SANITIZE:+ --sanitize $SANITIZE} $TEST_OPTIONS"
if [ "$configuration" = release ] && [ -z "$SHORT_TESTS" ]; then
test_flags="$test_flags -Xswiftc -DSYNCHRONIZATIONKIT_LONG_TESTS"
fi
Expand All @@ -596,11 +605,10 @@ jobs:
# is spelled out because, left to their defaults on the runner,
# avdmanager writes the AVD where the emulator does not look. `-cores`
# likewise: an AVD's default is fewer than the runner has, few enough
# that the measurements skip themselves for want of four, and that the
# long stress scale starves the tests that poll against a budget. The boot
# wait is the only wait, and bounded: an emulator that died on launch
# never becomes a device, and `adb wait-for-device` would sit out the
# job's whole timeout on it.
# that the long stress scale starves the tests that poll against a
# budget. The boot wait is the only wait, and bounded: an emulator that
# died on launch never becomes a device, and `adb wait-for-device`
# would sit out the job's whole timeout on it.
#
# The emulator outlives this step — the runner reaps stray processes
# when the job ends, not before — and adb goes on PATH for the step that
Expand Down Expand Up @@ -653,28 +661,30 @@ jobs:
# --skip-build` would, by hand: stage the bundle with every shared
# library it loads — the Swift runtime from the SDK, libc++ from the NDK
# — push the lot, and run it over adb, which hands back the exit status.
# Once for each testing library, because a runner runs one per
# invocation, and once per runner: Swift 6.3 builds a single bundle for
# the package, where 6.4's build system builds a runner for each test
# target. Named one at a time: a glob on one line runs the first runner
# with the rest as its arguments, which ran one target's tests, none of
# its XCTest cases, and succeeded. XCTest is left out of the debug run:
# its runner takes names and no pattern, and every XCTest case here is a
# measurement, which release runs whole. A runner for one target may hold
# no suite the selection names, and Swift Testing says so by exiting
# `EX_UNAVAILABLE`, 69, which is not a failure; `swift test` makes the
# same allowance on the other rows. A function called on the left of `||`
# runs without `-e`, so the steps a later one depends on say for
# themselves that the function ends with them.
# Once per runner: Swift 6.3 builds a single bundle for the package,
# where 6.4's build system builds a runner for each test target. Named
# one at a time: a glob on one line runs the first runner with the rest
# as its arguments, which ran one target's tests and succeeded. A
# runner for one target may hold no suite the selection names, and
# Swift Testing says so by exiting `EX_UNAVAILABLE`, 69, which is not a
# failure; `swift test` makes the same allowance on the other rows. A
# function called on the left of `||` runs without `-e`, so the steps
# a later one depends on say for themselves that the function ends
# with them.
#
# Swift Testing alone, everywhere, which the build flags above say with
# `--disable-xctest`: no XCTest case is left in the package — the
# measurements that were XCTest's live in the benchmark package now —
# and the corelibs runner traps on an empty list rather than reporting
# none.
- name: Run tests
if: ${{ !matrix.skip_tests }}
shell: bash
run: |
run_tests() {
local configuration="$1" test_flags="$2" select="$3" testable="" libraries=(swift-testing)
local configuration="$1" test_flags="$2" select="$3" testable=""
if [ "$configuration" = release ]; then
testable="--disable-testable-imports"
libraries=(xctest swift-testing)
fi

if [ -z "$ANDROID" ]; then
Expand All @@ -691,19 +701,15 @@ jobs:
cp -aL "$ANDROID_SWIFT_SDK/ndk-sysroot/usr/lib/$arch-linux-android/libc++_shared.so" "$staging" || return 1
adb push "$staging" "$device" || return 1

local runners=("$staging"/*-test-runner) runner library selection code status=0
local runners=("$staging"/*-test-runner) runner code status=0
if [ ! -e "${runners[0]}" ]; then runners=("$staging"/*.xctest); fi
if [ ! -e "${runners[0]}" ]; then echo "No test runner was built." >&2; return 1; fi
for runner in "${runners[@]}"; do
for library in "${libraries[@]}"; do
echo "$(basename "$runner") · $library"
selection=""
if [ "$library" = swift-testing ]; then selection="$select '$SUITES'"; fi
adb shell "cd $device && ./$(basename "$runner") --testing-library $library $selection" || {
code=$?
if [ "$library" != swift-testing ] || [ $code -ne 69 ]; then status=1; fi
}
done
echo "$(basename "$runner")"
adb shell "cd $device && ./$(basename "$runner") --testing-library swift-testing $select '$SUITES'" || {
code=$?
if [ $code -ne 69 ]; then status=1; fi
}
done
return $status
}
Expand All @@ -716,3 +722,20 @@ jobs:
run_tests release "$release_test_flags" --filter || status=1
echo "::endgroup::"
exit $status

# The benchmarks, on the rows that say so: one macOS and one Linux per
# minor, which is where the harness runs. They measure the library as
# it ships, built by the benchmark package's own manifest against the
# checkout above it, and put the table in the job's summary. Nothing
# fails on a number: a shared runner is noisy, and what the numbers are
# for is reading, here and against a baseline taken on one machine,
# which the README says how to do. `pipefail`, so that a benchmark that
# crashes fails the step through the tee.
- name: Run benchmarks
if: ${{ matrix.benchmarks }}
working-directory: Benchmarks
shell: bash
run: |
set -o pipefail
swift package benchmark --format markdown --no-progress --grouping metric | tee "$RUNNER_TEMP/benchmarks.md"
sed -n '/^## Baseline/,$p' "$RUNNER_TEMP/benchmarks.md" >> "$GITHUB_STEP_SUMMARY"
1 change: 1 addition & 0 deletions Benchmarks/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.benchmarkBaselines/
Loading
Loading