From 96ddb200fd53ce9de602f6dffbebb789a7861d48 Mon Sep 17 00:00:00 2001
From: Matthew Kocher <matthew.kocher@broadcom.com>
Date: Tue, 25 Aug 2026 02:37:38 -0700
Subject: [PATCH 1/4] Support Apple Silicon hosts in the builder image

Three changes needed to build stemcells on an arm64 Mac, where the
builder image and the rake tasks run under Rosetta x86-64 translation.

Adds an ARM64_TAR_FIX build arg that swaps the builder image's own tar
for the arm64 build. Ubuntu 26.04's tar (1.35+dfsg-4ubuntu0.4) cannot
extract anything under Rosetta -- every file creation fails with
"Cannot open: Function not implemented" (ENOSYS) -- while the arm64
build of the same version works because it runs natively on the arm64
kernel. It has to happen before anything else in the Dockerfile that
unpacks an archive: the ovftool bundle is self-extracting, and
ruby-install and syft are piped straight into tar. Extracting the
replacement deb itself needs a working tar, hence the temporary
busybox-static. Defaults to false and must stay false in CI, where
there is no arm64 emulation and the arm64 binary would not run at all.

Renumbers the build user to the host's USER_ID/GROUP_ID. The base image
already ships an `ubuntu` user at uid 1000, so on a host whose uid
differs -- macOS is typically 501-504 with gid 20 -- that user has to be
renumbered rather than created. Without this, running the container with
`--user "$(id -u):$(id -g)"` lands on a uid with no passwd entry, no
sudo, and no write access to the bind-mounted repo, /mnt/stemcells or
the gem home, and `gem install bundler` fails immediately.

Unpacks the monit source tarball outside the chroot in bosh_monit rather
than with `run_in_bosh_chroot ... tar zxvf`. The chroot's tar is the
target OS's x86-64 binary and hits the same ENOSYS; the builder
container's tar is known-good, and unpacking a source tarball needs no
chroot context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 .../os-image-stemcell-builder/Dockerfile      | 70 +++++++++++++++++--
 stemcell_builder/stages/bosh_monit/apply.sh   | 12 ++--
 2 files changed, 71 insertions(+), 11 deletions(-)

diff --git a/ci/docker/os-image-stemcell-builder/Dockerfile b/ci/docker/os-image-stemcell-builder/Dockerfile
index dd2a3b6a29..21799ad071 100644
--- a/ci/docker/os-image-stemcell-builder/Dockerfile
+++ b/ci/docker/os-image-stemcell-builder/Dockerfile
@@ -70,24 +70,80 @@ RUN apt-get update \
         xvfb \
     && locale-gen ${LANG}
 
+# Replace the x86-64 GNU tar with the arm64 build, for Apple Silicon hosts only.
+#
+# Ubuntu 26.04's tar (1.35+dfsg-4ubuntu0.4) cannot *extract* archives under
+# Rosetta x86-64 translation: every file creation fails with
+# "Cannot open: Function not implemented" (ENOSYS), because tar issues a
+# syscall Rosetta does not translate. Noble's tar is unaffected, and the arm64
+# build of the same version works because it runs natively on the arm64 kernel.
+#
+# This has to happen before anything else in this Dockerfile that unpacks an
+# archive — the ovftool bundle is self-extracting and the ruby-install and syft
+# downloads are piped straight into tar, so all of them fail without it. The
+# build proper shells out to tar just as heavily (the untar_base_os_image stage,
+# stemcell packaging, the RSpec helpers).
+#
+# dpkg unpacks debs with its own built-in tar reader rather than GNU tar, so apt
+# keeps working either way; only direct tar invocations are affected. Extracting
+# the arm64 deb itself needs a working tar, hence the temporary busybox.
+#
+# Leave this false when building on real x86-64 hardware (i.e. in CI): there is
+# no arm64 emulation there and the arm64 binary would not run at all.
+ARG ARM64_TAR_FIX=false
+RUN if [ "${ARM64_TAR_FIX}" = "true" ]; then \
+      set -eu \
+      && dpkg --add-architecture arm64 \
+      && apt-get update \
+      && apt-get install -y --no-install-recommends \
+           busybox-static \
+           libacl1:arm64 \
+           libc6:arm64 \
+           libselinux1:arm64 \
+      && mkdir -p /tmp/arm64-tar/root \
+      && cd /tmp/arm64-tar \
+      && apt-get download tar:arm64 \
+      && dpkg-deb --fsys-tarfile tar_*_arm64.deb | busybox tar -x -C /tmp/arm64-tar/root \
+      && mv /usr/bin/tar /usr/bin/tar.amd64 \
+      && install -m 0755 /tmp/arm64-tar/root/usr/bin/tar /usr/bin/tar \
+      && cd / \
+      && rm -rf /tmp/arm64-tar \
+      && apt-get purge -y busybox-static \
+      && apt-get autoremove -y \
+      && tar --version | head -1 ; \
+    fi
+
 # AppArmor's unix-chkpwd profile can block the Rosetta translator under
 # docker run --privileged on Apple Silicon; use a distinct helper name so PAM still works.
 RUN cp /usr/sbin/unix_chkpwd /usr/sbin/unix_chkpwd_rosetta \
   && chmod 4755 /usr/sbin/unix_chkpwd_rosetta \
   && ln -sf unix_chkpwd_rosetta /usr/sbin/unix_chkpwd
 
-RUN (id -u ubuntu &>/dev/null || useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu) \
-  && usermod -p '*' ubuntu \
-  && echo 'ubuntu ALL=NOPASSWD:ALL' >> /etc/sudoers
+# Give the build user the *host* uid/gid so that the directories below stay
+# writable when the container is started with `--user "$(id -u):$(id -g)"`.
+# The base image already ships an `ubuntu` user at uid 1000, so on hosts whose
+# uid differs (macOS is typically 501-504, gid 20) that user has to be
+# renumbered — a plain `useradd` would be skipped and the container would then
+# run as a uid with no passwd entry, no sudo, and no write access.
+RUN getent group ${GROUP_ID} >/dev/null || groupadd -g ${GROUP_ID} bosh
+RUN if getent passwd ${USER_ID} >/dev/null; then \
+      build_user="$(getent passwd ${USER_ID} | cut -d: -f1)"; \
+    elif id -u ubuntu >/dev/null 2>&1; then \
+      usermod -u ${USER_ID} -g ${GROUP_ID} ubuntu; build_user=ubuntu; \
+    else \
+      useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu; build_user=ubuntu; \
+    fi \
+  && usermod -p '*' "${build_user}" \
+  && echo "${build_user} ALL=NOPASSWD:ALL" >> /etc/sudoers
 
 RUN temp_dir="/mnt/tmp" \
     && mkdir -p "${temp_dir}" \
-    && chown -R ubuntu:ubuntu "${temp_dir}" \
-    && echo "export TMPDIR=${temp_dir}" >> ~ubuntu/.bashrc
+    && chown -R ${USER_ID}:${GROUP_ID} "${temp_dir}" \
+    && echo "export TMPDIR=${temp_dir}" >> /etc/bash.bashrc
 
 # rake tasks will be using this as chroot
 RUN mkdir -p /mnt/stemcells \
-    && chown -R ubuntu:ubuntu /mnt/stemcells
+    && chown -R ${USER_ID}:${GROUP_ID} /mnt/stemcells
 
 # VMware's ovftool is used to create vSphere stemcells
 ADD ${OVF_TOOL_INSTALLER} /tmp/ovftool_installer.bundle
@@ -121,7 +177,7 @@ RUN cd /tmp  \
       -- --disable-install-doc --disable-install-rdoc \
     && gem update --system \
     && mkdir -p "${GEM_HOME}/bin" \
-    && chown -R ubuntu:ubuntu "${GEM_HOME}"
+    && chown -R ${USER_ID}:${GROUP_ID} "${GEM_HOME}"
 
 RUN syft_cli_path="/usr/local/bin/syft" \
     && curl --show-error -sL "${SYFT_CLI_URL}" \
diff --git a/stemcell_builder/stages/bosh_monit/apply.sh b/stemcell_builder/stages/bosh_monit/apply.sh
index e8ed76903c..fde7b1d9c6 100755
--- a/stemcell_builder/stages/bosh_monit/apply.sh
+++ b/stemcell_builder/stages/bosh_monit/apply.sh
@@ -10,14 +10,18 @@ monit_basename=monit-5.2.5
 monit_archive=$monit_basename.tar.gz
 
 mkdir -p $chroot/$bosh_dir/src
-cp -r $dir/assets/$monit_archive $chroot/$bosh_dir/src
+
+# Unpack outside the chroot rather than with `run_in_bosh_chroot ... tar zxvf`.
+# The chroot's own tar is the target OS's binary, and Ubuntu 26.04's x86-64 tar
+# cannot extract under Rosetta (ENOSYS, see the ARM64_TAR_FIX note in
+# ci/docker/os-image-stemcell-builder/Dockerfile). The builder container's tar
+# is known-good, and unpacking a source tarball needs no chroot context.
+tar zxf $dir/assets/$monit_archive -C $chroot/$bosh_dir/src
 
 pkg_mgr install "zlib1g-dev libcrypt-dev"
 
 run_in_bosh_chroot $chroot "
-cd src
-tar zxvf $monit_archive
-cd $monit_basename
+cd src/$monit_basename
 ./configure --prefix=$bosh_dir --without-ssl CFLAGS='-fcommon' LIBS='-lcrypt'
 make -j4 && make install
 "

From c2ca2dc726dbef99b4e442dae63aff97cbca2410 Mon Sep 17 00:00:00 2001
From: Matthew Kocher <matthew.kocher@broadcom.com>
Date: Tue, 25 Aug 2026 01:56:06 -0700
Subject: [PATCH 2/4] Run arm64 unix_chkpwd, auditd and logrotate in rosetta
 stemcells

Replaces three x86-64 binaries with the arm64 builds of the same package
version, following the existing tar and systemd swaps, and removes the
workarounds that existed because they could not run under Rosetta.

unix_chkpwd: an AppArmor profile attached by path to
/{,usr/}{,s}bin/unix_chkpwd grants no access to the Rosetta interpreter,
which an x86-64 build needs in order to exec at all, so every PAM rule
that forks it failed. That broke `su` and rejected every `bosh ssh` at
the account stage right after the banner. It was worked around by
replacing /etc/pam.d/su and patching sshd's account stage to avoid
pam_unix; the su override also stopped verifying passwords altogether,
letting any wheel member become root without one. Both are gone and
/etc/pam.d is left as the packages and hardening stages produce it.

The interpreter cannot simply be allow-listed: AppArmor reports it as a
disconnected path, which needs flags=(attach_disconnected) on the
profile header and so cannot come from /etc/apparmor.d/local/.

auditd and logrotate: both units ship MemoryDenyWriteExecute=true, which
Rosetta's JIT cannot satisfy, and auditd additionally could not be
tracked by systemd under Type=forking + PIDFile. This removes the
rosetta-compat.conf drop-in and the auditd Type=simple/`auditd -n`
override.

rosetta-compat.conf disabled MemoryDenyWriteExecute, SystemCallFilter,
SystemCallArchitectures, LockPersonality and NoNewPrivileges on eight
units. None still need it: four are already arm64 systemd daemons,
systemd-timesyncd is not installed, systemd-udevd is masked by
base_warden, and auditd and logrotate are arm64 as of this change.
Removing it restores the upstream hardening rather than merely tidying
up, so rosetta_spec now asserts the overrides are absent.

Also stops masking ssh.socket. systemd derives an implicit
Requires=ssh.socket on ssh.service from the matching unit names, so
masking the socket made ssh.service unstartable, silently dropped the
job at boot, and left nothing listening on port 22 -- `bosh ssh` failed
with "connect to host ... port 22: Connection refused". The dependency
cannot be cleared from a drop-in; setting Sockets= does not remove it.
Socket activation works under Colima, so the override and its
RefuseManualStart=no drop-in are both removed.

Adds libauparse0t64:arm64 and libpopt0:arm64, needed by the arm64
auditd and logrotate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 bosh-stemcell/spec/stemcells/rosetta_spec.rb  | 105 ++++---
 .../base_ubuntu_warden_rosetta/apply.sh       | 263 +++++++++++-------
 .../assets/rosetta-compat.conf                |  12 -
 3 files changed, 223 insertions(+), 157 deletions(-)
 delete mode 100644 stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf

diff --git a/bosh-stemcell/spec/stemcells/rosetta_spec.rb b/bosh-stemcell/spec/stemcells/rosetta_spec.rb
index 86a8a25fbb..b666043f0b 100644
--- a/bosh-stemcell/spec/stemcells/rosetta_spec.rb
+++ b/bosh-stemcell/spec/stemcells/rosetta_spec.rb
@@ -43,65 +43,84 @@
     end
   end
 
-  context "Rosetta x86_64 emulation compatibility for Apple Silicon" do
-    # These systemd drop-in overrides disable security features that conflict
-    # with Rosetta's JIT compilation on Apple Silicon Macs.
+  context "arm64 tar (Rosetta extraction ENOSYS)" do
+    # Ubuntu 26.04's x86-64 GNU tar cannot extract archives under Rosetta: every
+    # file it creates fails with "Cannot open: Function not implemented"
+    # (ENOSYS). The BOSH agent shells out to tar for every release blob it
+    # downloads, so a stemcell shipping the x86-64 binary cannot run a single
+    # deployment. The stage swaps in the arm64 build and keeps the original as
+    # tar.amd64.
+    describe command("file -b /usr/bin/tar") do
+      its(:stdout) { should match(/ELF 64-bit.*ARM aarch64/) }
+    end
 
-    rosetta_services = %w[
-      systemd-journald
-      systemd-resolved
-      systemd-networkd
-      systemd-logind
-      systemd-timesyncd
-      auditd
-    ]
+    describe file("/usr/bin/tar.amd64") do
+      it { should be_file }
+    end
 
-    rosetta_services.each do |service|
-      describe file("/etc/systemd/system/#{service}.service.d/rosetta-compat.conf") do
+    # tar links against libacl and libselinux, so the arm64 builds of both have
+    # to be present or the binary will not even load.
+    %w[
+      /lib/aarch64-linux-gnu/libacl.so.1
+      /lib/aarch64-linux-gnu/libselinux.so.1
+    ].each do |lib|
+      describe file(lib) do
         it { should be_file }
-        its(:content) { should include("MemoryDenyWriteExecute=no") }
-        its(:content) { should include("LockPersonality=no") }
-        its(:content) { should include("NoNewPrivileges=no") }
       end
     end
 
-    describe file("/etc/systemd/system/systemd-binfmt.service") do
-      it { should be_linked_to File::NULL }
+    # The architecture check above would still pass if the binary could not
+    # actually run, so exercise a real round-trip.
+    describe command(
+      "set -e; " \
+      "rm -rf /tmp/tar-spec; mkdir -p /tmp/tar-spec/src /tmp/tar-spec/out; " \
+      "echo payload > /tmp/tar-spec/src/probe; " \
+      "tar -czf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec src; " \
+      "tar -xzf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec/out; " \
+      "grep -q payload /tmp/tar-spec/out/src/probe; " \
+      "rm -rf /tmp/tar-spec"
+    ) do
+      it("extracts an archive it just created") { expect(subject.exit_status).to eq(0) }
     end
   end
 
-  context "SSH without socket activation (Rosetta/Colima ENOSYS)" do
-    describe file("/etc/systemd/system/ssh.socket") do
-      it { should be_linked_to File::NULL }
+  context "Rosetta x86_64 emulation compatibility for Apple Silicon" do
+    # Asserted against the vendor units because these specs run on the built
+    # chroot, where `systemctl show` has no systemd to query.
+    describe file("/usr/lib/systemd/system/auditd.service") do
+      its(:content) { should match(/^MemoryDenyWriteExecute=true$/) }
     end
 
-    describe file("/etc/systemd/system/ssh.service.d/warden-no-socket-activation.conf") do
-      it { should be_file }
-      its(:content) { should include("RefuseManualStart=no") }
+    describe file("/usr/lib/systemd/system/logrotate.service") do
+      its(:content) { should match(/^MemoryDenyWriteExecute=true$/) }
     end
-  end
 
-  context "auditd foreground (Rosetta/Colima Docker pidfd ENOSYS)" do
-    # Under Docker/Colima with Rosetta emulation, systemd cannot create pidfd
-    # references or cgroup entries for processes started with Type=forking + PIDFile.
-    # Running auditd with -n (no-fork / foreground) avoids the fork-and-PIDFile
-    # lifecycle entirely, so systemd tracks the process directly without pidfd.
-    describe file("/etc/systemd/system/auditd.service.d/warden-auditd-foreground.conf") do
-      it { should be_file }
-      its(:content) { should include("Type=simple") }
-      its(:content) { should include("ExecStart=/usr/sbin/auditd -n") }
+    describe file("/etc/systemd/system/systemd-binfmt.service") do
+      it { should be_linked_to File::NULL }
     end
   end
 
-  context "restrict access to the su command CIS-9.5 (Rosetta PAM override)" do
-    # The Rosetta stemcell replaces /etc/pam.d/su with a minimal config that
-    # avoids unix-chkpwd (which AppArmor blocks under Lima/Rosetta, causing every
-    # su invocation to fail with "Authentication failure" even for root).
-    # pam_wheel.so use_uid is kept in the replacement config so that the CIS-9.5
-    # requirement — only wheel-group members may use su — is still enforced.
-    # This test verifies that the override did not inadvertently remove the wheel check.
-    describe command('grep "^\s*auth\s*required\s*pam_wheel.so\s*use_uid" /etc/pam.d/su') do
-      it("exits 0") { expect(subject.exit_status).to eq(0) }
+  context "arm64 userland binaries" do
+    # e_machine at offset 18 of the ELF header: 183 (EM_AARCH64), 62 for x86-64.
+    %w[
+      /usr/sbin/unix_chkpwd
+      /usr/sbin/auditd
+      /usr/sbin/logrotate
+    ].each do |path|
+      describe command("od -An -tu1 -j18 -N1 #{path} | tr -d ' '") do
+        its(:stdout) { should match(/^183$/) }
+      end
+
+      describe file("#{path}.amd64") do
+        it { should be_file }
+      end
+    end
+
+    # 0755, not the 2755 the deb ships: restrict_binary_setuid strips setgid
+    # outside the allowlist asserted in stemcells/ubuntu_spec.rb.
+    describe file("/usr/sbin/unix_chkpwd") do
+      it { should be_mode(0o755) }
+      its(:group) { should eq("shadow") }
     end
   end
 end
diff --git a/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh b/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh
index 5e714ae404..067e149042 100755
--- a/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh
+++ b/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh
@@ -1,32 +1,25 @@
 #!/usr/bin/env bash
 
-# Rosetta-specific warden stemcell stage.
+# Warden stemcell stage for running under Rosetta x86_64 emulation on an arm64
+# kernel (Colima/Lima on Apple Silicon).
 #
-# Applies two categories of fix for warden containers running under
-# Lima/Rosetta x86_64 emulation on Apple Silicon (arm64 kernel):
+# Rosetta does not translate every syscall, and its JIT needs writable+executable
+# memory, so some x86-64 binaries cannot run. Each is replaced with the arm64
+# build of the same package version, which runs natively on the arm64 kernel:
 #
-# 1. ARM64 SYSTEMD BINARIES
-#    On Apple Silicon (arm64 kernel + Rosetta x86_64 emulation), systemd v256+
-#    requires pidfd_open and pidfd_send_signal syscalls that Rosetta does not
-#    translate for x86_64 processes, causing ENOSYS. Replacing the systemd ELF
-#    binaries with arm64 equivalents causes them to run natively on the arm64
-#    kernel, giving full pidfd support.
+#   tar          cannot extract anything (ENOSYS)
+#   systemd      needs pidfd_open / pidfd_send_signal
+#   unix_chkpwd  AppArmor denies it the Rosetta interpreter, breaking PAM
+#   auditd       cannot satisfy Type=forking + MemoryDenyWriteExecute
+#   logrotate    killed by MemoryDenyWriteExecute
 #
-#    arm64 shared libraries land in /lib/aarch64-linux-gnu/ via Ubuntu multiarch
-#    and coexist with the x86_64 libraries already in /lib/x86_64-linux-gnu/.
-#    The arm64 ELF binaries reference those library paths via their built-in RPATH.
+# arm64 libraries install alongside the x86-64 ones under
+# /lib/aarch64-linux-gnu/ via multiarch; the arm64 binaries find them by RPATH.
 #
-# 2. ROSETTA COMPATIBILITY OVERRIDES
-#    Rosetta's JIT compiler requires writable+executable (W+X) memory.  Several
-#    systemd services have security hardening that blocks W+X (MemoryDenyWriteExecute,
-#    SystemCallFilter, etc.), causing them to crash on startup.  We apply drop-in
-#    overrides to disable these restrictions.  AppArmor also blocks unix-chkpwd
-#    from accessing the Rosetta runtime path, causing su to fail; we replace
-#    /etc/pam.d/su with a config that avoids unix-chkpwd entirely.
+# See docs/rosetta-stemcell-variant.md for the detail behind each entry.
 #
-# This stage is only inserted into warden_stages when the OS variant is
-# "rosetta" (i.e. rake ... ubuntu,resolute-rosetta,...). Standard warden
-# and all cloud infrastructure stemcell builds are unaffected.
+# Inserted into warden_stages only when the OS variant is "rosetta"
+# (rake ... ubuntu,resolute-rosetta,...); other builds are unaffected.
 
 set -e
 
@@ -34,18 +27,21 @@ base_dir=$(readlink -nf $(dirname $0)/../..)
 source $base_dir/lib/prelude_apply.bash
 
 # ---------------------------------------------------------------------------
-# Part 1: arm64 systemd binaries
+# Part 1: arm64 foreign architecture and runtime libraries
 # ---------------------------------------------------------------------------
 
 # Enable arm64 as a foreign architecture so apt can resolve arm64 packages.
 run_in_chroot $chroot "dpkg --add-architecture arm64"
 run_in_chroot $chroot "apt-get update"
 
-# Install arm64 runtime libraries that systemd depends on.
+# Install arm64 runtime libraries needed by the arm64 binaries installed below
+# (systemd in Part 3; unix_chkpwd, auditd and logrotate in Part 4 — auditd needs
+# libauparse, logrotate needs libpopt).
 # These install into /lib/aarch64-linux-gnu/ and /usr/lib/aarch64-linux-gnu/,
 # coexisting safely with the existing amd64 libraries.
 # libc6:arm64 also places /lib/ld-linux-aarch64.so.1 (the arm64 ELF interpreter).
 arm64_libs="libc6:arm64 \
+  libselinux1:arm64 \
   libcrypt1:arm64 \
   libgcrypt20:arm64 \
   libblkid1:arm64 \
@@ -61,10 +57,75 @@ arm64_libs="libc6:arm64 \
   libssl3t64:arm64 \
   libp11-kit0:arm64 \
   liblzma5:arm64 \
-  libgpg-error0:arm64"
+  libgpg-error0:arm64 \
+  libauparse0t64:arm64 \
+  libpopt0:arm64"
 
 run_in_chroot $chroot "apt-get install --no-install-recommends --assume-yes $arm64_libs"
 
+# ---------------------------------------------------------------------------
+# Part 2: arm64 tar
+# ---------------------------------------------------------------------------
+
+# This has to happen before the systemd debs below, because extracting those
+# uses the chroot's own tar — and Ubuntu 26.04's x86-64 GNU tar
+# (1.35+dfsg-4ubuntu0.4) cannot extract anything under Rosetta: every file it
+# creates fails with "Cannot open: Function not implemented" (ENOSYS), because
+# tar issues a syscall Rosetta does not translate. Ubuntu 24.04's tar is
+# unaffected, so this is new in 26.04. The arm64 build of the same version works
+# because it runs natively on the arm64 kernel. It is also the only tool in the
+# base image with this problem — gzip, xz, zstd, cpio, rsync and coreutils are
+# all fine.
+#
+# It matters at runtime too, not just during the build: the BOSH agent shells
+# out to tar to unpack every compiled package and release blob it downloads, so
+# a stemcell with a broken tar cannot run a single deployment.
+#
+# dpkg has its own built-in tar reader and does not call GNU tar to unpack debs,
+# so apt keeps working regardless; only direct tar invocations are affected
+# (including `dpkg -x`, which does shell out to GNU tar).
+#
+# `apt-get install tar:arm64` cannot be used: tar is Multi-Arch-incompatible and
+# tar:amd64 declares Conflicts: tar:arm64. Download and swap the binary in by
+# hand instead, the same way the systemd binaries are handled below.
+run_in_chroot $chroot "
+  mkdir -p /tmp/arm64-tar
+  cd /tmp/arm64-tar
+  apt-get download tar:arm64
+"
+
+# Unpack from outside the chroot: the chroot's own tar is the broken binary we
+# are about to replace, so it cannot be used to extract its own replacement.
+# The builder container's tar is known-good (see ARM64_TAR_FIX in
+# ci/docker/os-image-stemcell-builder/Dockerfile).
+mkdir -p "$chroot/tmp/arm64-tar/root"
+dpkg-deb --fsys-tarfile "$chroot"/tmp/arm64-tar/tar_*_arm64.deb \
+  | tar -x -C "$chroot/tmp/arm64-tar/root"
+
+# Keep the x86-64 binary as tar.amd64 for debugging and so the swap is obvious
+# to anyone inspecting the stemcell.
+mv "$chroot/usr/bin/tar" "$chroot/usr/bin/tar.amd64"
+install -m 0755 "$chroot/tmp/arm64-tar/root/usr/bin/tar" "$chroot/usr/bin/tar"
+rm -rf "$chroot/tmp/arm64-tar"
+
+# Fail loudly if the replacement cannot actually extract. Everything below
+# depends on it, and a silently broken tar would otherwise only surface much
+# later as an unexplained deployment failure.
+run_in_chroot $chroot "
+  set -e
+  rm -rf /tmp/tar-selftest
+  mkdir -p /tmp/tar-selftest/src /tmp/tar-selftest/out
+  echo rosetta > /tmp/tar-selftest/src/probe
+  tar -czf /tmp/tar-selftest/probe.tgz -C /tmp/tar-selftest src
+  tar -xzf /tmp/tar-selftest/probe.tgz -C /tmp/tar-selftest/out
+  grep -q rosetta /tmp/tar-selftest/out/src/probe
+  rm -rf /tmp/tar-selftest
+"
+
+# ---------------------------------------------------------------------------
+# Part 3: arm64 systemd binaries
+# ---------------------------------------------------------------------------
+
 # Download arm64 systemd debs without installing.
 # `apt-get install systemd:arm64` would conflict with systemd:amd64 because
 # systemd does not carry Multi-Arch: same. Download and extract manually instead.
@@ -114,86 +175,84 @@ run_in_chroot $chroot "
 run_in_chroot $chroot "rm -rf /tmp/arm64-debs"
 
 # ---------------------------------------------------------------------------
-# Part 2: Rosetta compatibility overrides
+# Part 4: arm64 userland binaries
 # ---------------------------------------------------------------------------
 
-# Apply systemd drop-ins that disable security hardening features conflicting
-# with Rosetta's JIT compilation (which requires writable+executable memory).
-rosetta_services=(
-  systemd-journald
-  systemd-resolved
-  systemd-networkd
-  systemd-logind
-  systemd-timesyncd
-  systemd-udevd
-  logrotate
-  auditd
-)
-
-for service in "${rosetta_services[@]}"; do
-  mkdir -p "$chroot/etc/systemd/system/${service}.service.d"
-  cp "$assets_dir/rosetta-compat.conf" "$chroot/etc/systemd/system/${service}.service.d/rosetta-compat.conf"
-done
-
-# Mask systemd-binfmt.service which fails under Rosetta emulation.
-run_in_chroot "$chroot" "systemctl mask systemd-binfmt.service"
-
-# auditd: systemd can return ENOSYS when creating pidfd/cgroup references from
-# PIDFile under Docker/Colima. Foreground auditd avoids forking + PIDFile quirks.
-mkdir -p "$chroot/etc/systemd/system/auditd.service.d"
-cat > "$chroot/etc/systemd/system/auditd.service.d/warden-auditd-foreground.conf" <<'UNIT'
-[Service]
-Type=simple
-PIDFile=
-ExecStart=
-ExecStart=/usr/sbin/auditd -n
-UNIT
-
-# Ubuntu enables OpenSSH via ssh.socket (socket activation). systemd's listener
-# stub fork can fail with ENOSYS under Docker/Colima with Rosetta x86_64
-# emulation. Use the traditional ssh.service so sshd binds port 22 itself.
-mkdir -p "$chroot/etc/systemd/system/ssh.service.d"
-cat > "$chroot/etc/systemd/system/ssh.service.d/warden-no-socket-activation.conf" <<'UNIT'
-[Unit]
-# When ssh.socket is masked, sshd must start via ssh.service; drop RefuseManualStart from the vendor unit.
-RefuseManualStart=no
-UNIT
-run_in_chroot "$chroot" "systemctl mask ssh.socket"
-run_in_chroot "$chroot" "systemctl enable ssh.service"
-
-# Fix `su` PAM authentication failures under Colima/Lima (Apple Silicon).
+# swap_arm64_binary <apt-package> <absolute-path> [group]
 #
-# Under Lima's Rosetta x86_64 emulation, AppArmor blocks unix-chkpwd
-# (an x86_64 binary) from accessing the Rosetta runtime path
-# /mnt/lima-rosetta/rosetta, causing every `su` invocation to fail with
-# "Authentication failure" — even for root.  The standard PAM config in
-# /etc/pam.d/su includes common-auth, which chains pam_faillock → pam_unix,
-# and pam_unix forks unix-chkpwd to verify passwords.  That fork triggers
-# the AppArmor denial.
+# None of these packages carry Multi-Arch: same, so `apt-get install <pkg>:arm64`
+# would conflict with the amd64 package; download and swap by hand instead. The
+# x86-64 build is kept as <path>.amd64, matching tar.amd64 above.
+swap_arm64_binary() {
+  local pkg="$1" path="$2" group="${3:-}"
+  local stage="/tmp/arm64-swap"
+
+  run_in_chroot $chroot "
+    set -e
+    rm -rf $stage
+    mkdir -p $stage
+    cd $stage
+    apt-get download ${pkg}:arm64
+    dpkg -x $stage/${pkg}_*_arm64.deb $stage/root
+  "
+
+  if [ ! -f "$chroot$stage/root$path" ]; then
+    echo "ERROR: ${pkg}:arm64 does not ship $path" >&2
+    exit 1
+  fi
+
+  mv "$chroot$path" "$chroot${path}.amd64"
+  install -m 0755 "$chroot$stage/root$path" "$chroot$path"
+  [ -n "$group" ] && run_in_chroot $chroot "chown root:$group $path"
+  run_in_chroot $chroot "rm -rf $stage"
+
+  # Fail loudly if the swap did not land. A silently-still-x86-64 binary would
+  # only resurface much later as an unexplained runtime failure.
+  run_in_chroot $chroot "
+    set -e
+    head -c 20 $path | grep -qa 'ELF' || { echo '$path is not an ELF binary' >&2; exit 1; }
+    # e_machine at offset 18 is 183 (EM_AARCH64) for arm64, 62 for x86-64.
+    arch_byte=\$(od -An -tu1 -j18 -N1 $path | tr -d ' ')
+    if [ \"\$arch_byte\" != '183' ]; then
+      echo \"ERROR: $path is not arm64 (e_machine=\$arch_byte)\" >&2
+      exit 1
+    fi
+  "
+}
+
+# pam_unix forks /usr/sbin/unix_chkpwd to read /etc/shadow. An AppArmor profile
+# attached by path to /{,usr/}{,s}bin/unix_chkpwd — loaded on the Lima VM and
+# enforced by the shared kernel, so it applies inside containers too — grants no
+# access to the Rosetta interpreter, which an x86-64 build needs in order to exec:
 #
-# pam_rootok.so already handles "root switching to any user" correctly on
-# its own, but because pam_unix / pam_faillock come after it in common-auth
-# they still run (pam_rootok is "sufficient" only when it *succeeds* at the
-# auth step level, not at the include level).  Using pam_permit for the
-# remaining auth/account rules means root can always su without unix-chkpwd.
+#   apparmor="DENIED" profile="unix-chkpwd" name="mnt/lima-rosetta/rosetta"
+#   pam_unix(sshd:account): unix_chkpwd abnormal exit: 5
 #
-# pam_wheel.so use_uid is retained (CIS-9.5): it only checks group membership
-# and never forks unix-chkpwd, so it is safe under Rosetta/AppArmor.
-# pam_rootok is "sufficient", so root bypasses the wheel check entirely;
-# non-root users must be in the wheel group (vcap is added by restrict_su_command).
+# The interpreter cannot be allow-listed: AppArmor reports it as a disconnected
+# path (note the missing leading slash), which needs flags=(attach_disconnected)
+# on the profile header and so cannot come from an /etc/apparmor.d/local/ include.
 #
-# This override is safe for warden containers: the host provides isolation,
-# and the container root user is already fully privileged.
-cat > "$chroot/etc/pam.d/su" <<'PAMEOF'
-# PAM configuration for su - warden Rosetta stemcell override.
-# AppArmor blocks unix-chkpwd under Lima/Rosetta causing su to fail even for root.
-# pam_rootok handles legitimate root → any-user su; pam_permit covers the rest.
-# pam_wheel.so use_uid is kept for CIS-9.5 compliance (does not invoke unix-chkpwd).
-auth       sufficient pam_rootok.so
-auth       required   pam_wheel.so use_uid
-auth       required   pam_permit.so
-account    required   pam_permit.so
-session    required   pam_env.so readenv=1
-session    required   pam_limits.so
-PAMEOF
-chmod 0644 "$chroot/etc/pam.d/su"
+# Mode stays 0755, not the 2755 the deb ships: restrict_binary_setuid strips
+# setgid outside the allowlist asserted in stemcells/ubuntu_spec.rb. su is
+# setuid-root and sshd is already privileged when they fork the helper, which is
+# how it reads /etc/shadow at 0400 root:root.
+swap_arm64_binary libpam-modules-bin /usr/sbin/unix_chkpwd shadow
+
+# auditd.service is Type=forking with a PIDFile and MemoryDenyWriteExecute=true;
+# an x86-64 build can satisfy neither under Rosetta. Only the daemon is swapped:
+# auditctl and augenrules cannot work in a container on any architecture, which
+# is why base_warden skips audit-rules.service.
+swap_arm64_binary auditd /usr/sbin/auditd
+
+# logrotate.service ships MemoryDenyWriteExecute=true, which kills the x86-64
+# binary outright (Result=signal).
+swap_arm64_binary logrotate /usr/sbin/logrotate
+
+# ---------------------------------------------------------------------------
+# Part 5: binfmt_misc
+# ---------------------------------------------------------------------------
+
+# systemd-binfmt rewrites /proc/sys/fs/binfmt_misc, where Lima registers the
+# Rosetta handler; letting it run risks deregistering Rosetta and leaving no
+# x86-64 binary in the container executable.
+run_in_chroot "$chroot" "systemctl mask systemd-binfmt.service"
diff --git a/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf b/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf
deleted file mode 100644
index 0f1912bd2b..0000000000
--- a/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf
+++ /dev/null
@@ -1,12 +0,0 @@
-[Service]
-# Disable security features that conflict with Rosetta x86_64 emulation
-# on Apple Silicon Macs. Rosetta uses JIT compilation which requires
-# writable+executable memory.
-#
-# These overrides are safe for warden stemcells since they run in
-# containerized environments where the host provides security isolation.
-MemoryDenyWriteExecute=no
-SystemCallArchitectures=
-SystemCallFilter=
-LockPersonality=no
-NoNewPrivileges=no

From b67f87ecb6f24aa0729d75bc100ec2b579f7051a Mon Sep 17 00:00:00 2001
From: Matthew Kocher <matthew.kocher@broadcom.com>
Date: Tue, 25 Aug 2026 01:56:22 -0700
Subject: [PATCH 3/4] Skip audit-rules and netplan-configure in warden
 containers

Both units failed permanently in every warden container, on every
architecture, which camouflaged real failures in `systemctl --failed`.

The kernel audit subsystem is not namespaced: audit_netlink_ok() in
kernel/audit.c rejects AUDIT_ADD_RULE, AUDIT_DEL_RULE and
AUDIT_LIST_RULES unless the caller is in the initial PID namespace. The
netlink socket opens and binds, then the operation returns -EPERM, so
audit-rules fails on the `-D` at line 2 of /etc/audit/audit.rules.
Being privileged and holding cap_audit_control does not help; only
--pid host does, which would defeat container isolation.

netplan-configure has no /etc/netplan to act on in these images, and its
ExecStartPost runs `udevadm control --reload` against the systemd-udevd
that this stage masks.

Skipped with ConditionVirtualization=!container rather than masked, so
the stemcell still behaves correctly if booted on a VM and the journal
records why. /etc/audit/audit.rules is untouched, so the STIG/CIS
content checks are unaffected, and auditd still starts because it only
Wants= the unit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 bosh-stemcell/spec/stemcells/warden_spec.rb  | 20 +++++++++++++++++++
 stemcell_builder/stages/base_warden/apply.sh | 21 ++++++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/bosh-stemcell/spec/stemcells/warden_spec.rb b/bosh-stemcell/spec/stemcells/warden_spec.rb
index 6fe3b9543b..77008131d7 100644
--- a/bosh-stemcell/spec/stemcells/warden_spec.rb
+++ b/bosh-stemcell/spec/stemcells/warden_spec.rb
@@ -29,4 +29,24 @@
       its(:content) { should match(/^kernel\.apparmor_restrict_unprivileged_unconfined = 0$/) }
     end
   end
+
+  context "units that cannot work in a container are skipped, not failed" do
+    # audit-rules needs the initial PID namespace; netplan-configure has no
+    # /etc/netplan and its ExecStartPost needs the masked systemd-udevd.
+    describe file("/etc/systemd/system/audit-rules.service.d/warden-skip-in-container.conf") do
+      it { should be_file }
+      its(:content) { should include("ConditionVirtualization=!container") }
+    end
+
+    describe file("/etc/systemd/system/netplan-configure.service.d/warden-skip-in-container.conf") do
+      it { should be_file }
+      its(:content) { should include("ConditionVirtualization=!container") }
+    end
+
+    # STIG/CIS checks read this file's content, so skipping the unit must not
+    # remove it.
+    describe file("/etc/audit/audit.rules") do
+      it { should be_file }
+    end
+  end
 end
diff --git a/stemcell_builder/stages/base_warden/apply.sh b/stemcell_builder/stages/base_warden/apply.sh
index ce367daded..abeb71484d 100755
--- a/stemcell_builder/stages/base_warden/apply.sh
+++ b/stemcell_builder/stages/base_warden/apply.sh
@@ -66,3 +66,24 @@ run_in_chroot "$chroot" "systemctl mask nvmf-autoconnect.service"
 
 # systemd-udevd has no function in containerised environments.
 run_in_chroot "$chroot" "systemctl mask systemd-udevd.service"
+
+# Audit rule changes need the initial PID namespace (audit_netlink_ok() in
+# kernel/audit.c returns -EPERM otherwise), so audit-rules.service can never
+# succeed in a container and would sit permanently failed. Skipped by condition
+# rather than masked so the stemcell still loads the rules if booted on a VM.
+# /etc/audit/audit.rules is left intact for the STIG/CIS content checks, and
+# auditd only Wants= this unit so it still starts.
+mkdir -p "$chroot/etc/systemd/system/audit-rules.service.d"
+cat > "$chroot/etc/systemd/system/audit-rules.service.d/warden-skip-in-container.conf" <<'UNIT'
+[Unit]
+ConditionVirtualization=!container
+UNIT
+
+# These images ship no /etc/netplan, so netplan-configure has nothing to
+# generate, and its ExecStartPost runs `udevadm control --reload` against the
+# systemd-udevd masked above.
+mkdir -p "$chroot/etc/systemd/system/netplan-configure.service.d"
+cat > "$chroot/etc/systemd/system/netplan-configure.service.d/warden-skip-in-container.conf" <<'UNIT'
+[Unit]
+ConditionVirtualization=!container
+UNIT

From 7db457b1ddfd5ee50c0444ec91664be7740326b2 Mon Sep 17 00:00:00 2001
From: Matthew Kocher <matthew.kocher@broadcom.com>
Date: Tue, 25 Aug 2026 01:56:44 -0700
Subject: [PATCH 4/4] Document Apple Silicon builds and the rosetta stemcell
 variant

Adds two docs for what were previously undocumented, and keeps them
apart because they are independent concerns: you can build a plain
warden stemcell on a Mac, and the -rosetta variant would be pointless on
an x86-64 host.

  docs/apple-silicon-builds.md      the build host: Colima setup, the
                                    builder image, the rake invocations,
                                    apt caching, and the gotchas
  docs/rosetta-stemcell-variant.md  the stemcell: what
                                    base_ubuntu_warden_rosetta changes
                                    and why each binary is replaced

The Quick Start's one-line `-rosetta` rake invocation moves into the
Apple Silicon doc, since it is unusable without the surrounding host
setup.

Corrects the ovftool note. ovftool is only used by the
image_ovf_generate stage, which appears solely in ovf_package_stages, so
warden builds never invoke it -- but the bundle is still required to
build the Docker image, because the Dockerfile ADDs it unconditionally.
The previous text said it was needed "when building a vSphere stemcell",
which understated it.

Dates the Ubuntu 26.04 tar breakage to tar 1.35+dfsg-4ubuntu0.4 as of
August 2026, with a note to recheck on a newer tar.

Also documents the USER_ID/GROUP_ID build args, that bosh-stemcell/ has
its own Gemfile, and how to run the warden and rosetta specs against a
stemcell built in a container.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 README.md                        |  81 ++++++++++++---
 docs/apple-silicon-builds.md     | 157 +++++++++++++++++++++++++++++
 docs/rosetta-stemcell-variant.md | 164 +++++++++++++++++++++++++++++++
 3 files changed, 391 insertions(+), 11 deletions(-)
 create mode 100644 docs/apple-silicon-builds.md
 create mode 100644 docs/rosetta-stemcell-variant.md

diff --git a/README.md b/README.md
index dd80810c53..0c3cca39d9 100644
--- a/README.md
+++ b/README.md
@@ -9,6 +9,18 @@ for 24.04).
 
 ## Quick Start: Building a Stemcell Locally
 
+Stemcells are always built as **x86-64**, regardless of your workstation's
+architecture. On an Apple Silicon Mac that means the whole build runs under
+Rosetta x86-64 translation, which needs a few extra steps — read
+[Building on Apple Silicon](docs/apple-silicon-builds.md) *instead of* this
+section if you are on an M-series Mac.
+
+Before you start, download `VMware-ovftool-*.bundle` into
+`ci/docker/os-image-stemcell-builder/`. The Docker image build `ADD`s it
+unconditionally, so it is required even though only vSphere and vCloud stemcells
+actually use `ovftool` — warden builds never invoke it. See
+[External Assets](#external-assets).
+
 ```bash
 export short_name="resolute"
 
@@ -19,6 +31,8 @@ mkdir -p tmp
 docker build \
    --platform linux/amd64 \
    --build-arg BASE_IMAGE="ubuntu:${short_name}" \
+   --build-arg USER_ID="$(id -u)" \
+   --build-arg GROUP_ID="$(id -g)" \
    --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \
    --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \
    --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \
@@ -53,15 +67,14 @@ bundle exec rake stemcell:build[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubu
 
  # build warden (BOSH Lite) stemcell
 bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000]
-
- # build warden rosetta stemcell (Apple Silicon / Colima / Lima)
-bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name}-rosetta,${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000]
 ```
 
-When building a vSphere stemcell, you must download `VMware-ovftool-*.bundle`
-and place it in the `ci/docker/os-image-stemcell-builder/` directory before
-running `docker build`. See [External Assets](#external-assets) for download
-instructions.
+`USER_ID` and `GROUP_ID` must match your host user, so that the bind-mounted
+repo, `/mnt/stemcells` and the gem home are all writable by the uid that
+`docker run --user` selects. Passing them is not optional on macOS: the base
+image already ships an `ubuntu` user at uid 1000, so without them the container
+runs as a uid with no passwd entry, no `sudo`, and no write access — `gem
+install bundler` fails immediately.
 
 ### OS image
 
@@ -140,6 +153,16 @@ export short_name="resolute"
 bosh upload-stemcell tmp/bosh-stemcell-0.0.8-vsphere-esxi-ubuntu-${short_name}-go_agent.tgz
 ```
 
+## Building on Apple Silicon
+
+An arm64 Mac builds x86-64 stemcells under Rosetta translation. Set up Colima,
+build the builder image with `ARM64_TAR_FIX=true`, then run the same rake tasks
+as above: see [Building on Apple Silicon](docs/apple-silicon-builds.md).
+
+Building *on* Apple Silicon is a separate concern from the `-rosetta` stemcell
+**variant**, which makes the stemcell you produce able to run under Rosetta
+itself: see [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md).
+
 ## Testing
 
 ### How to run tests for OS Images
@@ -184,6 +207,34 @@ spec/stemcells/stig_spec.rb \
 spec/stemcells/cis_spec.rb
 ```
 
+Note that `bosh-stemcell/` has its own `Gemfile` — `bundle install` at the repo
+root is not enough to run the specs directly, and skipping it fails with
+`Bundler::GemNotFound` for `fakefs` and `timecop`.
+
+For a warden (BOSH Lite) or rosetta stemcell the paths differ, and the specs
+must run in the same container that built the stemcell, since `/mnt/stemcells`
+lives inside it:
+
+```shell
+cd /opt/bosh/bosh-stemcell; \
+bundle install; \
+STEMCELL_IMAGE=/mnt/stemcells/warden/boshlite/ubuntu/work/work/warden-boshlite-ubuntu.raw \
+STEMCELL_WORKDIR=/mnt/stemcells/warden/boshlite/ubuntu/work/work \
+STEMCELL_INFRASTRUCTURE=warden \
+OS_NAME=ubuntu \
+OS_VERSION=resolute \
+bundle exec rspec -fd --tag ~exclude_on_warden \
+spec/stemcells/warden_spec.rb \
+spec/stemcells/rosetta_spec.rb
+```
+
+`spec/stemcells/rosetta_spec.rb` asserts the changes made by the `-rosetta`
+variant: that the replaced binaries are arm64 ELF, that the arm64 runtime
+libraries they need are present, that `tar` can complete a real
+create-then-extract round trip, and that no PAM or systemd-hardening override
+has been reintroduced. It runs automatically as part of a `-rosetta` stemcell
+build. See [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md).
+
 ### How to run tests for `ShelloutTypes`
 
 In pursuit of more robustly testing, we wrote our testing library for stemcell
@@ -250,10 +301,14 @@ If you find yourself debugging any of the above processes, here is what you need
 The installer for `ovftool` can be found at:
 - https://developer.broadcom.com/tools/open-virtualization-format-ovf-tool/latest.
 
-The `ovftool` installer **for linux** must be copied into 
+`ovftool` itself is only used by the `image_ovf_generate` stage, which appears
+only in `ovf_package_stages` — vSphere and vCloud. Warden stemcell builds never
+invoke it.
+
+The installer **for linux** must nonetheless be copied into
 [os-image-stemcell-builder](ci/docker/os-image-stemcell-builder)
-next to the `Dockerfile` before building the Docker image. If not you will
-see an error similar to: 
+next to the `Dockerfile` before building the Docker image, because the
+`Dockerfile` `ADD`s it unconditionally. If not you will see an error similar to: 
 
 ```shell
 ADD failed: failed to compute cache key: "/VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle": not found
@@ -266,7 +321,9 @@ The Docker image is published to
 You will need the ovftool installer present in
 `ci/docker/os-image-stemcell-builder/`.
 
-Rebuild the container with:
+Rebuild the container with the command below. On Apple Silicon add
+`--build-arg ARM64_TAR_FIX=true`; see
+[Building on Apple Silicon](docs/apple-silicon-builds.md).
 
 ```shell
 export short_name="resolute"
@@ -274,6 +331,8 @@ export short_name="resolute"
 docker build \
     --platform linux/amd64 \
     --build-arg BASE_IMAGE="ubuntu:${short_name}" \
+    --build-arg USER_ID="$(id -u)" \
+    --build-arg GROUP_ID="$(id -g)" \
     --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \
     --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \
     --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \
diff --git a/docs/apple-silicon-builds.md b/docs/apple-silicon-builds.md
new file mode 100644
index 0000000000..49c2ae2959
--- /dev/null
+++ b/docs/apple-silicon-builds.md
@@ -0,0 +1,157 @@
+# Building on Apple Silicon
+
+How to build stemcells on an arm64 Mac. This is about the **build host**: the
+builder image and the rake tasks are x86-64 and run under Rosetta translation.
+
+It is independent of the `-rosetta` stemcell *variant*, which is a property of
+the stemcell you produce — see [Rosetta stemcell variant](rosetta-stemcell-variant.md).
+You can build a plain `warden` stemcell here, and the variant only matters if you
+intend to run the resulting stemcell under Rosetta too.
+
+## Prerequisites
+
+Colima with the `vz` VM type and Rosetta enabled (`~/.colima/default/colima.yaml`):
+
+```yaml
+arch: aarch64
+vmType: vz
+rosetta: true
+mountType: virtiofs
+cpu: 16       # the build is long; give it what you can spare
+memory: 32
+disk: 64
+```
+
+Those are the values this was verified with. The build itself is more modest:
+about 6GB of scratch in `/mnt/stemcells`, a 3GB builder image, and roughly 1.2GB
+of artifacts in `tmp/`. Leave headroom for the BuildKit cache, which grows
+quickly across repeated attempts (`docker buildx du`, `docker buildx prune`).
+
+Check Rosetta is actually wired up before going further:
+
+```shell
+docker run --rm --platform linux/amd64 ubuntu:resolute uname -m   # => x86_64
+```
+
+`docker buildx ls` will *not* list `linux/amd64` among the builder's platforms.
+That is cosmetic: Colima's Rosetta binfmt handler is not advertised to BuildKit
+but is used anyway, and `docker build --platform linux/amd64` works.
+
+## Build the builder image
+
+```shell
+export short_name="resolute"
+
+docker build \
+   --platform linux/amd64 \
+   --build-arg BASE_IMAGE="ubuntu:${short_name}" \
+   --build-arg USER_ID="$(id -u)" \
+   --build-arg GROUP_ID="$(id -g)" \
+   --build-arg ARM64_TAR_FIX=true \
+   --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \
+   --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \
+   --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \
+   --build-arg RUBY_INSTALL_URL="https://github.com/postmodern/ruby-install/releases/download/v0.10.2/ruby-install-0.10.2.tar.gz" \
+   --build-arg RUBY_VERSION="$(cat .ruby-version)" \
+   --build-arg GEM_HOME="/usr/local/bundle" \
+   --build-arg OVF_TOOL_INSTALLER="VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle" \
+   --build-arg OVF_TOOL_INSTALLER_SHA1="6c24e473be49c961cfc3bb16774b52b48e822991" \
+   -t bosh/os-image-stemcell-builder:${short_name}-rosetta \
+   ci/docker/os-image-stemcell-builder/
+```
+
+`ARM64_TAR_FIX=true` is required here and must stay at its `false` default in
+CI, where there is no arm64 emulation and the arm64 binary would not run. It
+replaces the builder image's own `tar` — see
+[the tar problem](#ubuntu-2604s-x86-64-tar-does-not-work-under-rosetta) below.
+
+Expect five to ten minutes, most of it compiling Ruby under translation.
+
+### ovftool
+
+`ovftool` is only used by the `image_ovf_generate` stage, which appears solely in
+`ovf_package_stages` — vSphere and vCloud. **Warden stemcells never invoke it.**
+
+The `OVF_TOOL_INSTALLER` bundle is nonetheless required to build the *builder
+image*, because the Dockerfile `ADD`s it unconditionally. Download it into
+`ci/docker/os-image-stemcell-builder/` first. If you only ever build warden
+stemcells, making that `ADD` conditional would remove the dependency.
+
+## Build the OS image and the stemcell
+
+Start a long-lived container and do both builds in it. Gems live in the
+container's `/usr/local/bundle`, not in the bind mount, so a fresh `docker run`
+always needs `bundle install` again; keeping one container avoids that and keeps
+`/mnt/stemcells` around for re-running tests.
+
+```shell
+export short_name="resolute"
+
+docker run -it --name ${short_name}-rosetta-build \
+   --platform linux/amd64 \
+   --privileged \
+   -v "$(pwd):/opt/bosh" \
+   --workdir /opt/bosh \
+   --user="$(id -u):$(id -g)" \
+   bosh/os-image-stemcell-builder:${short_name}-rosetta
+
+# You're now in the Docker container
+export short_name="resolute"
+gem install bundler
+bundle install
+
+# build the OS image (the stemcell build below consumes it)
+bundle exec rake stemcell:build_os_image[ubuntu,${short_name},/opt/bosh/tmp/ubuntu_base_image_${short_name}.tgz]
+
+# build the warden rosetta stemcell
+bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name}-rosetta,/opt/bosh/tmp/ubuntu_base_image_${short_name}.tgz,9.000]
+```
+
+Each took roughly 10-15 minutes on a 16-CPU VM with a local apt cache; expect
+longer without one. Both run their RSpec suites at the end, so a clean exit
+means the tests passed too.
+
+The OS image is architecture-neutral and carries no Rosetta-specific changes.
+The `-rosetta` suffix on the *stemcell* build is what inserts the
+`base_ubuntu_warden_rosetta` stage, so the same OS image tarball also serves a
+plain `warden` build.
+
+### Rebuilding at the same version
+
+`create-env` records uploaded stemcells by name and version in its state file.
+Rebuilding without bumping the version leaves that record stale, and the next
+`create-env` reports `Skipped [Stemcell already uploaded]` and reuses the old
+image. Either bump the version or drop the `stemcells` entry (and
+`current_stemcell_id`) from `state.json`.
+
+## Speeding up repeat builds
+
+The build downloads well over a gigabyte of debs. If you run
+[apt-cacher-ng](https://hub.docker.com/r/sameersbn/apt-cacher-ng) on the host,
+add `-e http_proxy=http://host.docker.internal:3142` to `docker run` and both
+debootstrap and the in-chroot `apt` calls will use it.
+
+## Gotchas
+
+### Never use a `ROSETTA_` prefix for an env var or Docker `ARG`
+
+Rosetta reserves that namespace and refuses to start *any* process when it sees
+one it does not recognise, so the container dies instantly with `rosetta error:
+invalid ROSETTA_ environment variable ...` and exit code 133. Docker turns every
+`ARG` into an env var for `RUN`, which is why the tar build arg is called
+`ARM64_TAR_FIX`.
+
+### Ubuntu 26.04's x86-64 `tar` does not work under Rosetta
+
+Every extraction fails with `Cannot open: Function not implemented` (ENOSYS),
+because `tar` issues a syscall Rosetta does not translate.
+
+As of August 2026 this affects `tar` 1.35+dfsg-4ubuntu0.4 on Ubuntu 26.04.
+Ubuntu 24.04's `tar` is unaffected, so it is new in 26.04, and `tar` is the only
+affected tool in the base image — `gzip`, `xz`, `zstd`, `cpio`, `rsync` and
+coreutils all work. `apt` also works, because `dpkg` unpacks debs with its own
+built-in tar reader; only direct `tar` calls break, including `dpkg -x`. Recheck
+whether the workaround is still needed when moving to a newer `tar`.
+
+A bare `exit code 2` from a step that unpacks an archive is almost always this.
+Re-run the command by hand to see the `Function not implemented` lines.
diff --git a/docs/rosetta-stemcell-variant.md b/docs/rosetta-stemcell-variant.md
new file mode 100644
index 0000000000..541132310b
--- /dev/null
+++ b/docs/rosetta-stemcell-variant.md
@@ -0,0 +1,164 @@
+# The `-rosetta` stemcell variant
+
+A stemcell built as `...,resolute-rosetta,...` inserts the
+`base_ubuntu_warden_rosetta` stage into `warden_stages`. That stage makes the
+**resulting stemcell** able to run as a container under Rosetta x86_64 emulation
+on an arm64 kernel.
+
+This is separate from [building on Apple Silicon](apple-silicon-builds.md),
+which is about the build host. A plain `warden` stemcell built on a Mac is not a
+Rosetta stemcell, and this variant would be pointless on an x86-64 host.
+
+Only warden builds are affected; cloud infrastructure stemcells never include
+the stage. Because it is gated on the variant
+(`stage_collection.rb`, `operating_system.variant == "rosetta"`), the standard CI
+Resolute build never exercises it — changes here have to be verified against a
+deliberate `-rosetta` build.
+
+## Why binaries get replaced
+
+Rosetta does not translate every syscall, and its JIT compiler needs
+writable+executable memory. A handful of x86-64 binaries therefore cannot run at
+all. Each is replaced with the arm64 build of the same package version, which
+runs natively on the arm64 kernel.
+
+arm64 libraries install alongside the x86-64 ones under `/lib/aarch64-linux-gnu/`
+via Ubuntu multiarch, and the arm64 binaries find them through their built-in
+RPATH. Each replaced binary is kept as `<path>.amd64` so the swap is visible when
+inspecting a stemcell.
+
+| binary | why the x86-64 build fails |
+| --- | --- |
+| `tar` | Cannot extract anything (ENOSYS). See [the tar problem](apple-silicon-builds.md#ubuntu-2604s-x86-64-tar-does-not-work-under-rosetta). Needed at runtime too: the BOSH agent shells out to `tar` for every release blob. |
+| systemd daemons | systemd v256+ needs `pidfd_open` / `pidfd_send_signal`. |
+| `unix_chkpwd` | AppArmor denies it the Rosetta interpreter. |
+| `auditd` | Cannot satisfy `Type=forking` + `MemoryDenyWriteExecute=true`. |
+| `logrotate` | Killed by `MemoryDenyWriteExecute=true` (`Result=signal`). |
+
+Only the systemd *daemons* in `/usr/lib/systemd/` are swapped. The CLI tools in
+`/usr/bin` (`systemctl`, `journalctl`, `udevadm`) stay x86-64: they talk to PID 1
+over D-Bus and never call pidfd themselves, and keeping them x86-64 lets the
+build-time RSpec suite run them inside the x86-64 chroot.
+
+### unix_chkpwd and AppArmor
+
+This one is not a missing syscall. `pam_unix` forks `/usr/sbin/unix_chkpwd` to
+read `/etc/shadow`, and an AppArmor profile is attached by path to
+`/{,usr/}{,s}bin/unix_chkpwd`. The profile is loaded on the Lima VM, but AppArmor
+is enforced by the shared kernel, so it confines that path inside containers too.
+It grants the binary and `/etc/shadow` and nothing else — in particular no access
+to the Rosetta interpreter, which an x86-64 build needs in order to exec:
+
+```text
+apparmor="DENIED" operation="open" profile="unix-chkpwd"
+  name="mnt/lima-rosetta/rosetta" info="Failed name lookup - disconnected path"
+pam_unix(sshd:account): unix_chkpwd abnormal exit: 5
+```
+
+Every PAM rule that forks the helper then fails: `su` at the auth stage, and
+`bosh ssh` at the account stage, where sshd rejects the login immediately after
+printing the banner even though pubkey auth succeeded.
+
+The interpreter cannot be allow-listed. AppArmor reports it as a *disconnected*
+path — note the missing leading slash above — because it lives on a mount it
+cannot resolve into the host namespace, so ordinary path rules never match.
+Allowing it needs `flags=(attach_disconnected)` on the profile header, which
+cannot come from an `/etc/apparmor.d/local/` include and would mean every
+developer editing their own VM's vendor profile.
+
+Using the arm64 build removes the interpreter from the picture, so `/etc/pam.d`
+can be left exactly as the packages and hardening stages produce it.
+
+## Before changing anything PAM-related
+
+`unix_chkpwd` has bitten this repo more than once, in ways that are easy to
+misdiagnose. Worth reading before touching `/etc/pam.d`, `/etc/shadow` or the
+helper's permissions.
+
+**Do not route around `unix_chkpwd` in PAM.** Replacing an account rule with
+`pam_permit`, or adding `account sufficient pam_rootok.so`, resolves the symptom
+by skipping the entire account stack — including the `pam_faillock` rule this repo
+adds in `password_policies/assets/ubuntu/common-account.patch`, and account-expiry
+enforcement. On `/etc/pam.d/su` the equivalent shortcut on the *auth* stack also
+stops passwords being verified at all, letting any wheel member become root
+without one.
+
+**`su: Authentication failure` does not mean the auth stack failed.** Ubuntu's
+`/bin/su` is util-linux, not shadow, and it prints `pam_strerror()` for a failure
+of *either* `pam_authenticate` or `pam_acct_mgmt`. The account stack is the more
+common culprit, because that is where `pam_unix` forks `unix_chkpwd`.
+
+**`/etc/shadow` is mode `0400`, not the `0000` STIG V-38504 asks for.** In PAM
+1.7.0, which Resolute ships, `unix_chkpwd` links `libcap-ng` and drops
+`CAP_DAC_OVERRIDE` before opening `/etc/shadow`. At `0000 root:root` nothing can
+read it — not even root, and not with full `CapEff` — so `unix_chkpwd … chkexpiry`
+exits 9 and every `su` fails. `0400` still denies group and other, which is the
+intent of the control. PAM 1.5.3 (Noble) and 1.4.0 (Jammy) are unaffected. See
+`stemcell_builder/stages/base_file_permission/apply.sh`.
+
+**Restoring the helper's setgid bit is a red herring for that one.**
+`restrict_binary_setuid` in `stemcell_builder/lib/prelude_bosh.bash` does strip it,
+but `/etc/shadow` is `root:root`, so `setgid shadow` grants nothing; re-adding it
+leaves `su` broken. It is also why this stage installs the arm64 helper `0755`
+rather than the `2755` the deb ships — see the table above.
+
+**Do not reproduce shadow-permission problems on Apple Silicon.** On aarch64,
+`unix_chkpwd … chkexpiry` succeeds against a `0000` shadow and `su` works, so an
+aarch64 repro of that bug falsely passes. It only manifests on x86_64. (The
+AppArmor problem above is the opposite: it needs Rosetta to reproduce.)
+
+**Do not reorder `pam_wheel` in `/etc/pam.d/su`.** `restrict_su_command` appends
+`auth required pam_wheel.so use_uid` for CIS-9.5, which lands *after* the stock
+file's `auth sufficient pam_rootok.so`. That ordering is what lets root bypass the
+wheel check; moving the appended line up would force root through it.
+
+**Test against a fresh container.** Hand-edited PAM files in a long-lived debug
+container will make broken configurations look like working ones:
+
+```shell
+docker run --rm --entrypoint bash bosh.io/stemcells:img-<id> -c '…'
+```
+
+## systemd-binfmt is masked
+
+`systemd-binfmt` rewrites `/proc/sys/fs/binfmt_misc`, which is where Lima
+registers the Rosetta handler. Letting it run risks deregistering Rosetta and
+leaving no x86-64 binary in the container executable. This is unrelated to the
+architecture of the `systemd-binfmt` binary itself.
+
+## Service hardening
+
+The stage relaxes no systemd hardening. `MemoryDenyWriteExecute`,
+`SystemCallFilter`, `LockPersonality` and `NoNewPrivileges` are left as the
+vendor units ship them.
+
+Relaxing them is only ever correct for a service whose binary is still x86-64,
+because the reason to do so is Rosetta's need for W+X memory. Before adding such
+an override, check the binary's architecture:
+
+```shell
+od -An -tu1 -j18 -N1 /usr/sbin/logrotate   # 183 = arm64, 62 = x86-64
+```
+
+An arm64 binary does not need it, and adding one weakens the stemcell for no
+benefit. `bosh-stemcell/spec/stemcells/rosetta_spec.rb` asserts these overrides
+are absent.
+
+## Units that cannot work in a container
+
+Handled in `base_warden`, not this stage, because they apply to every warden
+stemcell regardless of architecture. Both are skipped with
+`ConditionVirtualization=!container` rather than masked, so the stemcell still
+behaves correctly if booted on a VM, and the journal records why:
+
+- **`audit-rules.service`** — the kernel audit subsystem is not namespaced.
+  `audit_netlink_ok()` in `kernel/audit.c` rejects `AUDIT_ADD_RULE`,
+  `AUDIT_DEL_RULE` and `AUDIT_LIST_RULES` unless the caller is in the initial PID
+  namespace, so `auditctl` gets `-EPERM` and the unit fails on the `-D` at line 2
+  of `/etc/audit/audit.rules`. Being privileged and holding
+  `cap_audit_control` does not help. `/etc/audit/audit.rules` is left intact for
+  the STIG/CIS content checks, and `auditd` still starts because it only
+  `Wants=` this unit. Do not expect audit records from inside a container.
+- **`netplan-configure.service`** — these images ship no `/etc/netplan`, so it
+  generates nothing, and its `ExecStartPost` runs `udevadm control --reload`
+  against the `systemd-udevd` that `base_warden` masks.