Improve rosetta support - builder and stemcell variant

README.md

@@ -9,6 +9,18 @@ for 24.04).
 
 ## Quick Start: Building a Stemcell Locally
 
+Stemcells are always built as **x86-64**, regardless of your workstation's
+architecture. On an Apple Silicon Mac that means the whole build runs under
+Rosetta x86-64 translation, which needs a few extra steps — read
+[Building on Apple Silicon](docs/apple-silicon-builds.md) *instead of* this
+section if you are on an M-series Mac.
+
+Before you start, download `VMware-ovftool-*.bundle` into
+`ci/docker/os-image-stemcell-builder/`. The Docker image build `ADD`s it
+unconditionally, so it is required even though only vSphere and vCloud stemcells
+actually use `ovftool` — warden builds never invoke it. See
+[External Assets](#external-assets).
+
 ```bash
 export short_name="resolute"
 
@@ -19,6 +31,8 @@ mkdir -p tmp
 docker build \
    --platform linux/amd64 \
    --build-arg BASE_IMAGE="ubuntu:${short_name}" \
+   --build-arg USER_ID="$(id -u)" \
+   --build-arg GROUP_ID="$(id -g)" \
    --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \
    --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \
    --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \
@@ -53,15 +67,14 @@ bundle exec rake stemcell:build[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubu
 
  # build warden (BOSH Lite) stemcell
 bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000]
-
- # build warden rosetta stemcell (Apple Silicon / Colima / Lima)
-bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name}-rosetta,${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000]
 ```
 
-When building a vSphere stemcell, you must download `VMware-ovftool-*.bundle`
-and place it in the `ci/docker/os-image-stemcell-builder/` directory before
-running `docker build`. See [External Assets](#external-assets) for download
-instructions.
+`USER_ID` and `GROUP_ID` must match your host user, so that the bind-mounted
+repo, `/mnt/stemcells` and the gem home are all writable by the uid that
+`docker run --user` selects. Passing them is not optional on macOS: the base
+image already ships an `ubuntu` user at uid 1000, so without them the container
+runs as a uid with no passwd entry, no `sudo`, and no write access — `gem
+install bundler` fails immediately.
 
 ### OS image
 
@@ -140,6 +153,16 @@ export short_name="resolute"
 bosh upload-stemcell tmp/bosh-stemcell-0.0.8-vsphere-esxi-ubuntu-${short_name}-go_agent.tgz
 ```
 
+## Building on Apple Silicon
+
+An arm64 Mac builds x86-64 stemcells under Rosetta translation. Set up Colima,
+build the builder image with `ARM64_TAR_FIX=true`, then run the same rake tasks
+as above: see [Building on Apple Silicon](docs/apple-silicon-builds.md).
+
+Building *on* Apple Silicon is a separate concern from the `-rosetta` stemcell
+**variant**, which makes the stemcell you produce able to run under Rosetta
+itself: see [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md).
+
 ## Testing
 
 ### How to run tests for OS Images
@@ -184,6 +207,34 @@ spec/stemcells/stig_spec.rb \
 spec/stemcells/cis_spec.rb
 ```
 
+Note that `bosh-stemcell/` has its own `Gemfile` — `bundle install` at the repo
+root is not enough to run the specs directly, and skipping it fails with
+`Bundler::GemNotFound` for `fakefs` and `timecop`.
+
+For a warden (BOSH Lite) or rosetta stemcell the paths differ, and the specs
+must run in the same container that built the stemcell, since `/mnt/stemcells`
+lives inside it:
+
+```shell
+cd /opt/bosh/bosh-stemcell; \
+bundle install; \
+STEMCELL_IMAGE=/mnt/stemcells/warden/boshlite/ubuntu/work/work/warden-boshlite-ubuntu.raw \
+STEMCELL_WORKDIR=/mnt/stemcells/warden/boshlite/ubuntu/work/work \
+STEMCELL_INFRASTRUCTURE=warden \
+OS_NAME=ubuntu \
+OS_VERSION=resolute \
+bundle exec rspec -fd --tag ~exclude_on_warden \
+spec/stemcells/warden_spec.rb \
+spec/stemcells/rosetta_spec.rb
+```
+
+`spec/stemcells/rosetta_spec.rb` asserts the changes made by the `-rosetta`
+variant: that the replaced binaries are arm64 ELF, that the arm64 runtime
+libraries they need are present, that `tar` can complete a real
+create-then-extract round trip, and that no PAM or systemd-hardening override
+has been reintroduced. It runs automatically as part of a `-rosetta` stemcell
+build. See [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md).
+
 ### How to run tests for `ShelloutTypes`
 
 In pursuit of more robustly testing, we wrote our testing library for stemcell
@@ -250,10 +301,14 @@ If you find yourself debugging any of the above processes, here is what you need
 The installer for `ovftool` can be found at:
 - https://developer.broadcom.com/tools/open-virtualization-format-ovf-tool/latest.
 
-The `ovftool` installer **for linux** must be copied into 
+`ovftool` itself is only used by the `image_ovf_generate` stage, which appears
+only in `ovf_package_stages` — vSphere and vCloud. Warden stemcell builds never
+invoke it.
+
+The installer **for linux** must nonetheless be copied into
 [os-image-stemcell-builder](ci/docker/os-image-stemcell-builder)
-next to the `Dockerfile` before building the Docker image. If not you will
-see an error similar to: 
+next to the `Dockerfile` before building the Docker image, because the
+`Dockerfile` `ADD`s it unconditionally. If not you will see an error similar to: 
 
 ```shell
 ADD failed: failed to compute cache key: "/VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle": not found
@@ -266,14 +321,18 @@ The Docker image is published to
 You will need the ovftool installer present in
 `ci/docker/os-image-stemcell-builder/`.
 
-Rebuild the container with:
+Rebuild the container with the command below. On Apple Silicon add
+`--build-arg ARM64_TAR_FIX=true`; see
+[Building on Apple Silicon](docs/apple-silicon-builds.md).
 
 ```shell
 export short_name="resolute"
 
 docker build \
     --platform linux/amd64 \
     --build-arg BASE_IMAGE="ubuntu:${short_name}" \
+    --build-arg USER_ID="$(id -u)" \
+    --build-arg GROUP_ID="$(id -g)" \
     --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \
     --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \
     --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \

bosh-stemcell/spec/stemcells/rosetta_spec.rb

@@ -43,65 +43,84 @@
     end
   end
 
-  context "Rosetta x86_64 emulation compatibility for Apple Silicon" do
-    # These systemd drop-in overrides disable security features that conflict
-    # with Rosetta's JIT compilation on Apple Silicon Macs.
+  context "arm64 tar (Rosetta extraction ENOSYS)" do
+    # Ubuntu 26.04's x86-64 GNU tar cannot extract archives under Rosetta: every
+    # file it creates fails with "Cannot open: Function not implemented"
+    # (ENOSYS). The BOSH agent shells out to tar for every release blob it
+    # downloads, so a stemcell shipping the x86-64 binary cannot run a single
+    # deployment. The stage swaps in the arm64 build and keeps the original as
+    # tar.amd64.
+    describe command("file -b /usr/bin/tar") do
+      its(:stdout) { should match(/ELF 64-bit.*ARM aarch64/) }
+    end
 
-    rosetta_services = %w[
-      systemd-journald
-      systemd-resolved
-      systemd-networkd
-      systemd-logind
-      systemd-timesyncd
-      auditd
-    ]
+    describe file("/usr/bin/tar.amd64") do
+      it { should be_file }
+    end
 
-    rosetta_services.each do |service|
-      describe file("/etc/systemd/system/#{service}.service.d/rosetta-compat.conf") do
+    # tar links against libacl and libselinux, so the arm64 builds of both have
+    # to be present or the binary will not even load.
+    %w[
+      /lib/aarch64-linux-gnu/libacl.so.1
+      /lib/aarch64-linux-gnu/libselinux.so.1
+    ].each do |lib|
+      describe file(lib) do
         it { should be_file }
-        its(:content) { should include("MemoryDenyWriteExecute=no") }
-        its(:content) { should include("LockPersonality=no") }
-        its(:content) { should include("NoNewPrivileges=no") }
       end
     end
 
-    describe file("/etc/systemd/system/systemd-binfmt.service") do
-      it { should be_linked_to File::NULL }
+    # The architecture check above would still pass if the binary could not
+    # actually run, so exercise a real round-trip.
+    describe command(
+      "set -e; " \
+      "rm -rf /tmp/tar-spec; mkdir -p /tmp/tar-spec/src /tmp/tar-spec/out; " \
+      "echo payload > /tmp/tar-spec/src/probe; " \
+      "tar -czf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec src; " \
+      "tar -xzf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec/out; " \
+      "grep -q payload /tmp/tar-spec/out/src/probe; " \
+      "rm -rf /tmp/tar-spec"
+    ) do
+      it("extracts an archive it just created") { expect(subject.exit_status).to eq(0) }
     end
   end
 
-  context "SSH without socket activation (Rosetta/Colima ENOSYS)" do
-    describe file("/etc/systemd/system/ssh.socket") do
-      it { should be_linked_to File::NULL }
+  context "Rosetta x86_64 emulation compatibility for Apple Silicon" do
+    # Asserted against the vendor units because these specs run on the built
+    # chroot, where `systemctl show` has no systemd to query.
+    describe file("/usr/lib/systemd/system/auditd.service") do
+      its(:content) { should match(/^MemoryDenyWriteExecute=true$/) }
     end
 
-    describe file("/etc/systemd/system/ssh.service.d/warden-no-socket-activation.conf") do
-      it { should be_file }
-      its(:content) { should include("RefuseManualStart=no") }
+    describe file("/usr/lib/systemd/system/logrotate.service") do
+      its(:content) { should match(/^MemoryDenyWriteExecute=true$/) }
     end
-  end
 
-  context "auditd foreground (Rosetta/Colima Docker pidfd ENOSYS)" do
-    # Under Docker/Colima with Rosetta emulation, systemd cannot create pidfd
-    # references or cgroup entries for processes started with Type=forking + PIDFile.
-    # Running auditd with -n (no-fork / foreground) avoids the fork-and-PIDFile
-    # lifecycle entirely, so systemd tracks the process directly without pidfd.
-    describe file("/etc/systemd/system/auditd.service.d/warden-auditd-foreground.conf") do
-      it { should be_file }
-      its(:content) { should include("Type=simple") }
-      its(:content) { should include("ExecStart=/usr/sbin/auditd -n") }
+    describe file("/etc/systemd/system/systemd-binfmt.service") do
+      it { should be_linked_to File::NULL }
     end
   end
 
-  context "restrict access to the su command CIS-9.5 (Rosetta PAM override)" do
-    # The Rosetta stemcell replaces /etc/pam.d/su with a minimal config that
-    # avoids unix-chkpwd (which AppArmor blocks under Lima/Rosetta, causing every
-    # su invocation to fail with "Authentication failure" even for root).
-    # pam_wheel.so use_uid is kept in the replacement config so that the CIS-9.5
-    # requirement — only wheel-group members may use su — is still enforced.
-    # This test verifies that the override did not inadvertently remove the wheel check.
-    describe command('grep "^\s*auth\s*required\s*pam_wheel.so\s*use_uid" /etc/pam.d/su') do
-      it("exits 0") { expect(subject.exit_status).to eq(0) }
+  context "arm64 userland binaries" do
+    # e_machine at offset 18 of the ELF header: 183 (EM_AARCH64), 62 for x86-64.
+    %w[
+      /usr/sbin/unix_chkpwd
+      /usr/sbin/auditd
+      /usr/sbin/logrotate
+    ].each do |path|
+      describe command("od -An -tu1 -j18 -N1 #{path} | tr -d ' '") do
+        its(:stdout) { should match(/^183$/) }
+      end
+
+      describe file("#{path}.amd64") do
+        it { should be_file }
+      end
+    end
+
+    # 0755, not the 2755 the deb ships: restrict_binary_setuid strips setgid
+    # outside the allowlist asserted in stemcells/ubuntu_spec.rb.
+    describe file("/usr/sbin/unix_chkpwd") do
+      it { should be_mode(0o755) }
+      its(:group) { should eq("shadow") }
     end
   end
 end

bosh-stemcell/spec/stemcells/warden_spec.rb

@@ -29,4 +29,24 @@
       its(:content) { should match(/^kernel\.apparmor_restrict_unprivileged_unconfined = 0$/) }
     end
   end
+
+  context "units that cannot work in a container are skipped, not failed" do
+    # audit-rules needs the initial PID namespace; netplan-configure has no
+    # /etc/netplan and its ExecStartPost needs the masked systemd-udevd.
+    describe file("/etc/systemd/system/audit-rules.service.d/warden-skip-in-container.conf") do
+      it { should be_file }
+      its(:content) { should include("ConditionVirtualization=!container") }
+    end
+
+    describe file("/etc/systemd/system/netplan-configure.service.d/warden-skip-in-container.conf") do
+      it { should be_file }
+      its(:content) { should include("ConditionVirtualization=!container") }
+    end
+
+    # STIG/CIS checks read this file's content, so skipping the unit must not
+    # remove it.
+    describe file("/etc/audit/audit.rules") do
+      it { should be_file }
+    end
+  end
 end

ci/docker/os-image-stemcell-builder/Dockerfile

@@ -70,24 +70,80 @@ RUN apt-get update \
         xvfb \
     && locale-gen ${LANG}
 
+# Replace the x86-64 GNU tar with the arm64 build, for Apple Silicon hosts only.
+#
+# Ubuntu 26.04's tar (1.35+dfsg-4ubuntu0.4) cannot *extract* archives under
+# Rosetta x86-64 translation: every file creation fails with
+# "Cannot open: Function not implemented" (ENOSYS), because tar issues a
+# syscall Rosetta does not translate. Noble's tar is unaffected, and the arm64
+# build of the same version works because it runs natively on the arm64 kernel.
+#
+# This has to happen before anything else in this Dockerfile that unpacks an
+# archive — the ovftool bundle is self-extracting and the ruby-install and syft
+# downloads are piped straight into tar, so all of them fail without it. The
+# build proper shells out to tar just as heavily (the untar_base_os_image stage,
+# stemcell packaging, the RSpec helpers).
+#
+# dpkg unpacks debs with its own built-in tar reader rather than GNU tar, so apt
+# keeps working either way; only direct tar invocations are affected. Extracting
+# the arm64 deb itself needs a working tar, hence the temporary busybox.
+#
+# Leave this false when building on real x86-64 hardware (i.e. in CI): there is
+# no arm64 emulation there and the arm64 binary would not run at all.
+ARG ARM64_TAR_FIX=false
+RUN if [ "${ARM64_TAR_FIX}" = "true" ]; then \
+      set -eu \
+      && dpkg --add-architecture arm64 \
+      && apt-get update \
+      && apt-get install -y --no-install-recommends \
+           busybox-static \
+           libacl1:arm64 \
+           libc6:arm64 \
+           libselinux1:arm64 \
+      && mkdir -p /tmp/arm64-tar/root \
+      && cd /tmp/arm64-tar \
+      && apt-get download tar:arm64 \
+      && dpkg-deb --fsys-tarfile tar_*_arm64.deb | busybox tar -x -C /tmp/arm64-tar/root \
+      && mv /usr/bin/tar /usr/bin/tar.amd64 \
+      && install -m 0755 /tmp/arm64-tar/root/usr/bin/tar /usr/bin/tar \
+      && cd / \
+      && rm -rf /tmp/arm64-tar \
+      && apt-get purge -y busybox-static \
+      && apt-get autoremove -y \
+      && tar --version | head -1 ; \
+    fi
+
 # AppArmor's unix-chkpwd profile can block the Rosetta translator under
 # docker run --privileged on Apple Silicon; use a distinct helper name so PAM still works.
 RUN cp /usr/sbin/unix_chkpwd /usr/sbin/unix_chkpwd_rosetta \
   && chmod 4755 /usr/sbin/unix_chkpwd_rosetta \
   && ln -sf unix_chkpwd_rosetta /usr/sbin/unix_chkpwd
 
-RUN (id -u ubuntu &>/dev/null || useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu) \
-  && usermod -p '*' ubuntu \
-  && echo 'ubuntu ALL=NOPASSWD:ALL' >> /etc/sudoers
+# Give the build user the *host* uid/gid so that the directories below stay
+# writable when the container is started with `--user "$(id -u):$(id -g)"`.
+# The base image already ships an `ubuntu` user at uid 1000, so on hosts whose
+# uid differs (macOS is typically 501-504, gid 20) that user has to be
+# renumbered — a plain `useradd` would be skipped and the container would then
+# run as a uid with no passwd entry, no sudo, and no write access.
+RUN getent group ${GROUP_ID} >/dev/null || groupadd -g ${GROUP_ID} bosh
+RUN if getent passwd ${USER_ID} >/dev/null; then \
+      build_user="$(getent passwd ${USER_ID} | cut -d: -f1)"; \
+    elif id -u ubuntu >/dev/null 2>&1; then \
+      usermod -u ${USER_ID} -g ${GROUP_ID} ubuntu; build_user=ubuntu; \
+    else \
+      useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu; build_user=ubuntu; \
+    fi \
+  && usermod -p '*' "${build_user}" \
+  && echo "${build_user} ALL=NOPASSWD:ALL" >> /etc/sudoers
 
 RUN temp_dir="/mnt/tmp" \
     && mkdir -p "${temp_dir}" \
-    && chown -R ubuntu:ubuntu "${temp_dir}" \
-    && echo "export TMPDIR=${temp_dir}" >> ~ubuntu/.bashrc
+    && chown -R ${USER_ID}:${GROUP_ID} "${temp_dir}" \
+    && echo "export TMPDIR=${temp_dir}" >> /etc/bash.bashrc
 
 # rake tasks will be using this as chroot
 RUN mkdir -p /mnt/stemcells \
-    && chown -R ubuntu:ubuntu /mnt/stemcells
+    && chown -R ${USER_ID}:${GROUP_ID} /mnt/stemcells
 
 # VMware's ovftool is used to create vSphere stemcells
 ADD ${OVF_TOOL_INSTALLER} /tmp/ovftool_installer.bundle
@@ -121,7 +177,7 @@ RUN cd /tmp  \
       -- --disable-install-doc --disable-install-rdoc \
     && gem update --system \
     && mkdir -p "${GEM_HOME}/bin" \
-    && chown -R ubuntu:ubuntu "${GEM_HOME}"
+    && chown -R ${USER_ID}:${GROUP_ID} "${GEM_HOME}"
 
 RUN syft_cli_path="/usr/local/bin/syft" \
     && curl --show-error -sL "${SYFT_CLI_URL}" \