From 8b52634c4a33867b05193551c4c55b93ef2ff150 Mon Sep 17 00:00:00 2001 From: Tobias Oberstein Date: Sun, 18 Oct 2026 15:03:52 +0200 Subject: [PATCH 1/2] start new dev branch; add audit file --- .audit/oberstet_fix_1940.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 .audit/oberstet_fix_1940.md diff --git a/.audit/oberstet_fix_1940.md b/.audit/oberstet_fix_1940.md new file mode 100644 index 000000000..55362e07a --- /dev/null +++ b/.audit/oberstet_fix_1940.md @@ -0,0 +1,8 @@ +- [ ] I did **not** use any AI-assistance tools to help create this pull request. +- [x] I **did** use AI-assistance tools to *help* create this pull request. +- [x] I have read, understood and followed the project's AI_POLICY.md when creating code, documentation etc. for this pull request. + +Submitted by: @oberstet +Date: 2026-10-18 +Related issue(s): #1940 +Branch: oberstet:fix_1940 From 8c1cb8baa87fefa1f1968206a0fb8fa02f51ad2c Mon Sep 17 00:00:00 2001 From: Tobias Oberstein Date: Sun, 18 Oct 2026 15:15:20 +0200 Subject: [PATCH 2/2] #1940: CI harness - build the ARM64 wheels natively on ubuntu-24.04-arm, not under QEMU The 10 ARM64 wheel builds ran on ubuntu-latest (x86_64) inside the target containers under QEMU user-mode emulation. In the last 5 completed runs (2026-10-18) a build took 37-40 minutes median and up to 96 minutes, and PyPy builds have segfaulted under emulation (exit 139), hence the 9 attempts of 120 minutes each (#1947). The build job now runs on ubuntu-24.04-arm, with the same container images, Dockerfiles and build script - the same switch as zlmdb #183. The tonistiigi/binfmt, setup-qemu-action and qemu-user-static steps are removed (the qemu-user-static image is x86_64 only). Buildx stays, it builds the custom PyPy images. A new first step fails if the runner is not aarch64. `--platform linux/arm64` stays and now only asserts the image architecture. The retry wrapper stays, with 60 minutes and 3 attempts as in zlmdb, until native runs have been measured. This replaces the A/B trial proposed in #1940: zlmdb's native jobs since #183 (16 runs, 2026-10-15..17, 288 ubuntu-24.04-arm jobs) were all successful, with a median pickup wait of 4 s (p95 288 s, max 595 s) and a median job duration of 1.7 minutes. Not changed here (follow-ups): the QEMU workarounds in the PyPy Dockerfiles (PYPY_DISABLE_JIT etc.) and in build-arm64-wheel.sh (nh3, py-ubjson), and native smoke tests of the built wheels (zlmdb #171). Harness change only, no red/green pair. Not verifiable locally: asgard1 is x86_64, and an emulated build would be QEMU again. The YAML parses; ruff check and just check-typing cpy314 pass. Expected CI result: GREEN - all 10 ARM64 builds green in a few minutes each with no retries, wheels and file names unchanged; main, wheels, wheels-docker and wstest stay green. Note: This work was completed with AI assistance (Claude Code). --- .github/scripts/build-arm64-wheel.sh | 2 +- .github/workflows/wheels-arm64.yml | 70 ++++++++++------------------ docs/changelog.rst | 1 + 3 files changed, 27 insertions(+), 46 deletions(-) diff --git a/.github/scripts/build-arm64-wheel.sh b/.github/scripts/build-arm64-wheel.sh index 3ea4048b4..e07df2a09 100755 --- a/.github/scripts/build-arm64-wheel.sh +++ b/.github/scripts/build-arm64-wheel.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Build ARM64 wheels inside Docker container with QEMU emulation +# Build ARM64 wheels inside Docker container on a native ARM64 runner (#1940) # This script runs INSIDE the manylinux ARM64 container set -ex diff --git a/.github/workflows/wheels-arm64.yml b/.github/workflows/wheels-arm64.yml index 127ed44e2..da219513a 100644 --- a/.github/workflows/wheels-arm64.yml +++ b/.github/workflows/wheels-arm64.yml @@ -27,10 +27,13 @@ jobs: # IMPORTANT: we still need .cicd as a Git submodule in the using repo though! # because e.g. identifiers.yml wants to access scripts/sanitize.sh ! + # Built natively on GitHub-hosted ARM64 runners, not under QEMU on x86_64 (#1940): + # under emulation builds segfaulted intermittently (PyPy exit 139) and took up + # to 96 minutes. Same switch as zlmdb #183. build-wheels: name: Build ARM64 wheels (${{ matrix.target.name }}) needs: identifiers - runs-on: ubuntu-latest + runs-on: ubuntu-24.04-arm env: BASE_REPO: ${{ needs.identifiers.outputs.base_repo }} @@ -158,47 +161,29 @@ jobs: with: submodules: recursive - # ============================================================ - # QEMU + Docker Buildx Setup - # ============================================================ - - - name: Install modern QEMU (tonistiigi/binfmt) + - name: Verify the runner is native ARM64 run: | - echo "==> Installing modern QEMU 8.x via tonistiigi/binfmt" - echo "This provides much better stability for PyPy on ARM64 emulation" - docker run --rm --privileged tonistiigi/binfmt --install all - echo "✅ Modern QEMU installed" - - - name: Set up QEMU for ARM64 emulation - uses: docker/setup-qemu-action@v3 - with: - platforms: linux/arm64 + echo "==> Runner architecture: $(uname -m)" + test "$(uname -m)" = "aarch64" || { echo "ERROR: expected a native aarch64 runner (#1940)"; exit 1; } + # Buildx builds the custom PyPy images below (natively, no emulation) - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - - name: Verify QEMU setup - run: | - echo "==> QEMU platforms available:" - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes - docker buildx ls - # ============================================================ # Build custom PyPy images (only for build_type == custom) # ============================================================ - - name: Build custom PyPy manylinux image (with retry for QEMU flakiness) + - name: Build custom PyPy manylinux image (with retry) if: matrix.target.build_type == 'custom' uses: nick-fields/retry@v3 with: - # NVX (CFFI/flatc) AND autobahn's Rust base deps (cbor2 6.x, cryptography) - # all source-compile under QEMU ARM64 emulation — none ship a PyPy-aarch64 - # wheel, so pip builds them from source, which is pathologically slow under - # emulation (a full run can exceed 60 min). Generous limits here are the - # interim, low-risk mitigation until we move ARM64 builds to native - # ubuntu-24.04-arm runners (removes emulation entirely). See #1947 and #1940. - timeout_minutes: 120 - max_attempts: 9 + # Under QEMU these builds took up to 96 minutes and segfaulted + # intermittently, hence 9 attempts of 120 minutes (#1947). Natively, + # keep the retry wrapper (as zlmdb #183) until native runs have been + # measured; then reduce (#1940). A timeout is not retried (retry_on: error). + timeout_minutes: 60 + max_attempts: 3 retry_on: error warning_on_retry: true command: | @@ -217,17 +202,15 @@ jobs: # Build wheels inside ARM64 containers # ============================================================ - - name: Build ARM64 wheels with NVX extension (with retry for QEMU flakiness) + - name: Build ARM64 wheels with NVX extension (with retry) uses: nick-fields/retry@v3 with: - # NVX (CFFI/flatc) AND autobahn's Rust base deps (cbor2 6.x, cryptography) - # all source-compile under QEMU ARM64 emulation — none ship a PyPy-aarch64 - # wheel, so pip builds them from source, which is pathologically slow under - # emulation (a full run can exceed 60 min). Generous limits here are the - # interim, low-risk mitigation until we move ARM64 builds to native - # ubuntu-24.04-arm runners (removes emulation entirely). See #1947 and #1940. - timeout_minutes: 120 - max_attempts: 9 + # Under QEMU these builds took up to 96 minutes and segfaulted + # intermittently, hence 9 attempts of 120 minutes (#1947). Natively, + # keep the retry wrapper (as zlmdb #183) until native runs have been + # measured; then reduce (#1940). A timeout is not retried (retry_on: error). + timeout_minutes: 60 + max_attempts: 3 retry_on: error warning_on_retry: true command: | @@ -248,7 +231,8 @@ jobs: # Create wheelhouse directory on host mkdir -p wheelhouse - # Run build script inside ARM64 container via QEMU + # Run build script inside the ARM64 container (natively; the + # platform flag only asserts the image architecture) docker run --rm \ --platform linux/arm64 \ -v $PWD:/io \ @@ -462,7 +446,7 @@ jobs: echo "Platform Tag: ${{ matrix.target.manylinux_tag }}" >> $BUILD_INFO echo "glibc Version: ${{ matrix.target.glibc_version }}" >> $BUILD_INFO echo "Python Implementation: ${{ matrix.target.python_impl }}" >> $BUILD_INFO - echo "Build Method: GitHub Actions + QEMU + Docker (ARM64 emulation)" >> $BUILD_INFO + echo "Build Method: GitHub Actions native ARM64 runner (ubuntu-24.04-arm) + Docker" >> $BUILD_INFO echo "NVX Acceleration: ENABLED (binary wheels with native extensions)" >> $BUILD_INFO echo "" >> $BUILD_INFO @@ -472,10 +456,6 @@ jobs: echo "- $(basename "$whl")" >> $BUILD_INFO done - echo "" >> $BUILD_INFO - echo "Note: These wheels were built using QEMU emulation on x86_64 runners." >> $BUILD_INFO - echo " They are fully functional native ARM64 binaries." >> $BUILD_INFO - echo "" echo "==> Generated build-info.txt:" cat $BUILD_INFO diff --git a/docs/changelog.rst b/docs/changelog.rst index 937a4ad5e..9e24fdd3a 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -21,6 +21,7 @@ Changelog * Run the unit tests (Twisted/trial and asyncio/pytest) in CI on every supported Python: CPython 3.11–3.14 and PyPy 3.11, instead of on CPython 3.14 only, and with all four compression extensions installed (``python-snappy`` is added to the ``dev`` extra; ``just check-compressors`` verifies all four are available) (#1986) * Adopt the contribution workflow shared by all WAMP projects: ``CONTRIBUTING.md``, the pull request template and ``.audit/README.md`` are now deployed byte-identically from wamp-cicd (GitHub issue first, red → green tests, AI-assistance disclosure) and kept in sync by a CI drift check; everything specific to Autobahn|Python moved to a new ``DEVELOPMENT.md`` (also included in the docs). ``.cicd`` is pinned to wamp-cicd ``38ded6a``, the same commit across the WAMP fleet (#1956) * Remove the opt-in entropy depletion tests (``autobahn/test/test_rng.py``, enabled by ``AUTOBAHN_CI_ENABLE_RNG_DEPLETION_TESTS``, never run in CI): they read ``/proc/sys/kernel/random/entropy_avail``, which has been a constant 256 since Linux 5.18, and ``/dev/random`` no longer blocks since Linux 5.6, so one test passed whatever happened and the other always failed (#1278) +* Build the ARM64 Linux wheels natively on GitHub's ``ubuntu-24.04-arm`` runners instead of under QEMU emulation on x86_64 runners, as zlmdb does since #183. Under emulation the builds took around 40 minutes (up to 96) and segfaulted intermittently, so each build had 9 attempts of 120 minutes; the same container images and build script are used, and a build attempt now gets 60 minutes and 3 tries (#1940) * Add an ``import-smoke-lowest`` CI job (``just test-imports-lowest``) that runs the import smoke test on CPython 3.11 with every direct dependency resolved to the lowest version ``pyproject.toml`` allows (``uv --resolution lowest-direct``), so stale version floors are caught. The import smoke test now also covers the Twisted and asyncio bindings, each module imported in a fresh interpreter (#1955) **asyncio**