diff --git a/README.md b/README.md index dd80810c53..0c3cca39d9 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,18 @@ for 24.04). ## Quick Start: Building a Stemcell Locally +Stemcells are always built as **x86-64**, regardless of your workstation's +architecture. On an Apple Silicon Mac that means the whole build runs under +Rosetta x86-64 translation, which needs a few extra steps — read +[Building on Apple Silicon](docs/apple-silicon-builds.md) *instead of* this +section if you are on an M-series Mac. + +Before you start, download `VMware-ovftool-*.bundle` into +`ci/docker/os-image-stemcell-builder/`. The Docker image build `ADD`s it +unconditionally, so it is required even though only vSphere and vCloud stemcells +actually use `ovftool` — warden builds never invoke it. See +[External Assets](#external-assets). + ```bash export short_name="resolute" @@ -19,6 +31,8 @@ mkdir -p tmp docker build \ --platform linux/amd64 \ --build-arg BASE_IMAGE="ubuntu:${short_name}" \ + --build-arg USER_ID="$(id -u)" \ + --build-arg GROUP_ID="$(id -g)" \ --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \ --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \ --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \ @@ -53,15 +67,14 @@ bundle exec rake stemcell:build[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubu # build warden (BOSH Lite) stemcell bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000] - - # build warden rosetta stemcell (Apple Silicon / Colima / Lima) -bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name}-rosetta,${PWD}/tmp/ubuntu_base_image_${short_name}.tgz,9.000] ``` -When building a vSphere stemcell, you must download `VMware-ovftool-*.bundle` -and place it in the `ci/docker/os-image-stemcell-builder/` directory before -running `docker build`. See [External Assets](#external-assets) for download -instructions. +`USER_ID` and `GROUP_ID` must match your host user, so that the bind-mounted +repo, `/mnt/stemcells` and the gem home are all writable by the uid that +`docker run --user` selects. Passing them is not optional on macOS: the base +image already ships an `ubuntu` user at uid 1000, so without them the container +runs as a uid with no passwd entry, no `sudo`, and no write access — `gem +install bundler` fails immediately. ### OS image @@ -140,6 +153,16 @@ export short_name="resolute" bosh upload-stemcell tmp/bosh-stemcell-0.0.8-vsphere-esxi-ubuntu-${short_name}-go_agent.tgz ``` +## Building on Apple Silicon + +An arm64 Mac builds x86-64 stemcells under Rosetta translation. Set up Colima, +build the builder image with `ARM64_TAR_FIX=true`, then run the same rake tasks +as above: see [Building on Apple Silicon](docs/apple-silicon-builds.md). + +Building *on* Apple Silicon is a separate concern from the `-rosetta` stemcell +**variant**, which makes the stemcell you produce able to run under Rosetta +itself: see [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md). + ## Testing ### How to run tests for OS Images @@ -184,6 +207,34 @@ spec/stemcells/stig_spec.rb \ spec/stemcells/cis_spec.rb ``` +Note that `bosh-stemcell/` has its own `Gemfile` — `bundle install` at the repo +root is not enough to run the specs directly, and skipping it fails with +`Bundler::GemNotFound` for `fakefs` and `timecop`. + +For a warden (BOSH Lite) or rosetta stemcell the paths differ, and the specs +must run in the same container that built the stemcell, since `/mnt/stemcells` +lives inside it: + +```shell +cd /opt/bosh/bosh-stemcell; \ +bundle install; \ +STEMCELL_IMAGE=/mnt/stemcells/warden/boshlite/ubuntu/work/work/warden-boshlite-ubuntu.raw \ +STEMCELL_WORKDIR=/mnt/stemcells/warden/boshlite/ubuntu/work/work \ +STEMCELL_INFRASTRUCTURE=warden \ +OS_NAME=ubuntu \ +OS_VERSION=resolute \ +bundle exec rspec -fd --tag ~exclude_on_warden \ +spec/stemcells/warden_spec.rb \ +spec/stemcells/rosetta_spec.rb +``` + +`spec/stemcells/rosetta_spec.rb` asserts the changes made by the `-rosetta` +variant: that the replaced binaries are arm64 ELF, that the arm64 runtime +libraries they need are present, that `tar` can complete a real +create-then-extract round trip, and that no PAM or systemd-hardening override +has been reintroduced. It runs automatically as part of a `-rosetta` stemcell +build. See [The `-rosetta` stemcell variant](docs/rosetta-stemcell-variant.md). + ### How to run tests for `ShelloutTypes` In pursuit of more robustly testing, we wrote our testing library for stemcell @@ -250,10 +301,14 @@ If you find yourself debugging any of the above processes, here is what you need The installer for `ovftool` can be found at: - https://developer.broadcom.com/tools/open-virtualization-format-ovf-tool/latest. -The `ovftool` installer **for linux** must be copied into +`ovftool` itself is only used by the `image_ovf_generate` stage, which appears +only in `ovf_package_stages` — vSphere and vCloud. Warden stemcell builds never +invoke it. + +The installer **for linux** must nonetheless be copied into [os-image-stemcell-builder](ci/docker/os-image-stemcell-builder) -next to the `Dockerfile` before building the Docker image. If not you will -see an error similar to: +next to the `Dockerfile` before building the Docker image, because the +`Dockerfile` `ADD`s it unconditionally. If not you will see an error similar to: ```shell ADD failed: failed to compute cache key: "/VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle": not found @@ -266,7 +321,9 @@ The Docker image is published to You will need the ovftool installer present in `ci/docker/os-image-stemcell-builder/`. -Rebuild the container with: +Rebuild the container with the command below. On Apple Silicon add +`--build-arg ARM64_TAR_FIX=true`; see +[Building on Apple Silicon](docs/apple-silicon-builds.md). ```shell export short_name="resolute" @@ -274,6 +331,8 @@ export short_name="resolute" docker build \ --platform linux/amd64 \ --build-arg BASE_IMAGE="ubuntu:${short_name}" \ + --build-arg USER_ID="$(id -u)" \ + --build-arg GROUP_ID="$(id -g)" \ --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \ --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \ --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \ diff --git a/bosh-stemcell/spec/stemcells/rosetta_spec.rb b/bosh-stemcell/spec/stemcells/rosetta_spec.rb index 86a8a25fbb..b666043f0b 100644 --- a/bosh-stemcell/spec/stemcells/rosetta_spec.rb +++ b/bosh-stemcell/spec/stemcells/rosetta_spec.rb @@ -43,65 +43,84 @@ end end - context "Rosetta x86_64 emulation compatibility for Apple Silicon" do - # These systemd drop-in overrides disable security features that conflict - # with Rosetta's JIT compilation on Apple Silicon Macs. + context "arm64 tar (Rosetta extraction ENOSYS)" do + # Ubuntu 26.04's x86-64 GNU tar cannot extract archives under Rosetta: every + # file it creates fails with "Cannot open: Function not implemented" + # (ENOSYS). The BOSH agent shells out to tar for every release blob it + # downloads, so a stemcell shipping the x86-64 binary cannot run a single + # deployment. The stage swaps in the arm64 build and keeps the original as + # tar.amd64. + describe command("file -b /usr/bin/tar") do + its(:stdout) { should match(/ELF 64-bit.*ARM aarch64/) } + end - rosetta_services = %w[ - systemd-journald - systemd-resolved - systemd-networkd - systemd-logind - systemd-timesyncd - auditd - ] + describe file("/usr/bin/tar.amd64") do + it { should be_file } + end - rosetta_services.each do |service| - describe file("/etc/systemd/system/#{service}.service.d/rosetta-compat.conf") do + # tar links against libacl and libselinux, so the arm64 builds of both have + # to be present or the binary will not even load. + %w[ + /lib/aarch64-linux-gnu/libacl.so.1 + /lib/aarch64-linux-gnu/libselinux.so.1 + ].each do |lib| + describe file(lib) do it { should be_file } - its(:content) { should include("MemoryDenyWriteExecute=no") } - its(:content) { should include("LockPersonality=no") } - its(:content) { should include("NoNewPrivileges=no") } end end - describe file("/etc/systemd/system/systemd-binfmt.service") do - it { should be_linked_to File::NULL } + # The architecture check above would still pass if the binary could not + # actually run, so exercise a real round-trip. + describe command( + "set -e; " \ + "rm -rf /tmp/tar-spec; mkdir -p /tmp/tar-spec/src /tmp/tar-spec/out; " \ + "echo payload > /tmp/tar-spec/src/probe; " \ + "tar -czf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec src; " \ + "tar -xzf /tmp/tar-spec/probe.tgz -C /tmp/tar-spec/out; " \ + "grep -q payload /tmp/tar-spec/out/src/probe; " \ + "rm -rf /tmp/tar-spec" + ) do + it("extracts an archive it just created") { expect(subject.exit_status).to eq(0) } end end - context "SSH without socket activation (Rosetta/Colima ENOSYS)" do - describe file("/etc/systemd/system/ssh.socket") do - it { should be_linked_to File::NULL } + context "Rosetta x86_64 emulation compatibility for Apple Silicon" do + # Asserted against the vendor units because these specs run on the built + # chroot, where `systemctl show` has no systemd to query. + describe file("/usr/lib/systemd/system/auditd.service") do + its(:content) { should match(/^MemoryDenyWriteExecute=true$/) } end - describe file("/etc/systemd/system/ssh.service.d/warden-no-socket-activation.conf") do - it { should be_file } - its(:content) { should include("RefuseManualStart=no") } + describe file("/usr/lib/systemd/system/logrotate.service") do + its(:content) { should match(/^MemoryDenyWriteExecute=true$/) } end - end - context "auditd foreground (Rosetta/Colima Docker pidfd ENOSYS)" do - # Under Docker/Colima with Rosetta emulation, systemd cannot create pidfd - # references or cgroup entries for processes started with Type=forking + PIDFile. - # Running auditd with -n (no-fork / foreground) avoids the fork-and-PIDFile - # lifecycle entirely, so systemd tracks the process directly without pidfd. - describe file("/etc/systemd/system/auditd.service.d/warden-auditd-foreground.conf") do - it { should be_file } - its(:content) { should include("Type=simple") } - its(:content) { should include("ExecStart=/usr/sbin/auditd -n") } + describe file("/etc/systemd/system/systemd-binfmt.service") do + it { should be_linked_to File::NULL } end end - context "restrict access to the su command CIS-9.5 (Rosetta PAM override)" do - # The Rosetta stemcell replaces /etc/pam.d/su with a minimal config that - # avoids unix-chkpwd (which AppArmor blocks under Lima/Rosetta, causing every - # su invocation to fail with "Authentication failure" even for root). - # pam_wheel.so use_uid is kept in the replacement config so that the CIS-9.5 - # requirement — only wheel-group members may use su — is still enforced. - # This test verifies that the override did not inadvertently remove the wheel check. - describe command('grep "^\s*auth\s*required\s*pam_wheel.so\s*use_uid" /etc/pam.d/su') do - it("exits 0") { expect(subject.exit_status).to eq(0) } + context "arm64 userland binaries" do + # e_machine at offset 18 of the ELF header: 183 (EM_AARCH64), 62 for x86-64. + %w[ + /usr/sbin/unix_chkpwd + /usr/sbin/auditd + /usr/sbin/logrotate + ].each do |path| + describe command("od -An -tu1 -j18 -N1 #{path} | tr -d ' '") do + its(:stdout) { should match(/^183$/) } + end + + describe file("#{path}.amd64") do + it { should be_file } + end + end + + # 0755, not the 2755 the deb ships: restrict_binary_setuid strips setgid + # outside the allowlist asserted in stemcells/ubuntu_spec.rb. + describe file("/usr/sbin/unix_chkpwd") do + it { should be_mode(0o755) } + its(:group) { should eq("shadow") } end end end diff --git a/bosh-stemcell/spec/stemcells/warden_spec.rb b/bosh-stemcell/spec/stemcells/warden_spec.rb index 6fe3b9543b..77008131d7 100644 --- a/bosh-stemcell/spec/stemcells/warden_spec.rb +++ b/bosh-stemcell/spec/stemcells/warden_spec.rb @@ -29,4 +29,24 @@ its(:content) { should match(/^kernel\.apparmor_restrict_unprivileged_unconfined = 0$/) } end end + + context "units that cannot work in a container are skipped, not failed" do + # audit-rules needs the initial PID namespace; netplan-configure has no + # /etc/netplan and its ExecStartPost needs the masked systemd-udevd. + describe file("/etc/systemd/system/audit-rules.service.d/warden-skip-in-container.conf") do + it { should be_file } + its(:content) { should include("ConditionVirtualization=!container") } + end + + describe file("/etc/systemd/system/netplan-configure.service.d/warden-skip-in-container.conf") do + it { should be_file } + its(:content) { should include("ConditionVirtualization=!container") } + end + + # STIG/CIS checks read this file's content, so skipping the unit must not + # remove it. + describe file("/etc/audit/audit.rules") do + it { should be_file } + end + end end diff --git a/ci/docker/os-image-stemcell-builder/Dockerfile b/ci/docker/os-image-stemcell-builder/Dockerfile index dd2a3b6a29..21799ad071 100644 --- a/ci/docker/os-image-stemcell-builder/Dockerfile +++ b/ci/docker/os-image-stemcell-builder/Dockerfile @@ -70,24 +70,80 @@ RUN apt-get update \ xvfb \ && locale-gen ${LANG} +# Replace the x86-64 GNU tar with the arm64 build, for Apple Silicon hosts only. +# +# Ubuntu 26.04's tar (1.35+dfsg-4ubuntu0.4) cannot *extract* archives under +# Rosetta x86-64 translation: every file creation fails with +# "Cannot open: Function not implemented" (ENOSYS), because tar issues a +# syscall Rosetta does not translate. Noble's tar is unaffected, and the arm64 +# build of the same version works because it runs natively on the arm64 kernel. +# +# This has to happen before anything else in this Dockerfile that unpacks an +# archive — the ovftool bundle is self-extracting and the ruby-install and syft +# downloads are piped straight into tar, so all of them fail without it. The +# build proper shells out to tar just as heavily (the untar_base_os_image stage, +# stemcell packaging, the RSpec helpers). +# +# dpkg unpacks debs with its own built-in tar reader rather than GNU tar, so apt +# keeps working either way; only direct tar invocations are affected. Extracting +# the arm64 deb itself needs a working tar, hence the temporary busybox. +# +# Leave this false when building on real x86-64 hardware (i.e. in CI): there is +# no arm64 emulation there and the arm64 binary would not run at all. +ARG ARM64_TAR_FIX=false +RUN if [ "${ARM64_TAR_FIX}" = "true" ]; then \ + set -eu \ + && dpkg --add-architecture arm64 \ + && apt-get update \ + && apt-get install -y --no-install-recommends \ + busybox-static \ + libacl1:arm64 \ + libc6:arm64 \ + libselinux1:arm64 \ + && mkdir -p /tmp/arm64-tar/root \ + && cd /tmp/arm64-tar \ + && apt-get download tar:arm64 \ + && dpkg-deb --fsys-tarfile tar_*_arm64.deb | busybox tar -x -C /tmp/arm64-tar/root \ + && mv /usr/bin/tar /usr/bin/tar.amd64 \ + && install -m 0755 /tmp/arm64-tar/root/usr/bin/tar /usr/bin/tar \ + && cd / \ + && rm -rf /tmp/arm64-tar \ + && apt-get purge -y busybox-static \ + && apt-get autoremove -y \ + && tar --version | head -1 ; \ + fi + # AppArmor's unix-chkpwd profile can block the Rosetta translator under # docker run --privileged on Apple Silicon; use a distinct helper name so PAM still works. RUN cp /usr/sbin/unix_chkpwd /usr/sbin/unix_chkpwd_rosetta \ && chmod 4755 /usr/sbin/unix_chkpwd_rosetta \ && ln -sf unix_chkpwd_rosetta /usr/sbin/unix_chkpwd -RUN (id -u ubuntu &>/dev/null || useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu) \ - && usermod -p '*' ubuntu \ - && echo 'ubuntu ALL=NOPASSWD:ALL' >> /etc/sudoers +# Give the build user the *host* uid/gid so that the directories below stay +# writable when the container is started with `--user "$(id -u):$(id -g)"`. +# The base image already ships an `ubuntu` user at uid 1000, so on hosts whose +# uid differs (macOS is typically 501-504, gid 20) that user has to be +# renumbered — a plain `useradd` would be skipped and the container would then +# run as a uid with no passwd entry, no sudo, and no write access. +RUN getent group ${GROUP_ID} >/dev/null || groupadd -g ${GROUP_ID} bosh +RUN if getent passwd ${USER_ID} >/dev/null; then \ + build_user="$(getent passwd ${USER_ID} | cut -d: -f1)"; \ + elif id -u ubuntu >/dev/null 2>&1; then \ + usermod -u ${USER_ID} -g ${GROUP_ID} ubuntu; build_user=ubuntu; \ + else \ + useradd -u ${USER_ID} -g ${GROUP_ID} -m ubuntu; build_user=ubuntu; \ + fi \ + && usermod -p '*' "${build_user}" \ + && echo "${build_user} ALL=NOPASSWD:ALL" >> /etc/sudoers RUN temp_dir="/mnt/tmp" \ && mkdir -p "${temp_dir}" \ - && chown -R ubuntu:ubuntu "${temp_dir}" \ - && echo "export TMPDIR=${temp_dir}" >> ~ubuntu/.bashrc + && chown -R ${USER_ID}:${GROUP_ID} "${temp_dir}" \ + && echo "export TMPDIR=${temp_dir}" >> /etc/bash.bashrc # rake tasks will be using this as chroot RUN mkdir -p /mnt/stemcells \ - && chown -R ubuntu:ubuntu /mnt/stemcells + && chown -R ${USER_ID}:${GROUP_ID} /mnt/stemcells # VMware's ovftool is used to create vSphere stemcells ADD ${OVF_TOOL_INSTALLER} /tmp/ovftool_installer.bundle @@ -121,7 +177,7 @@ RUN cd /tmp \ -- --disable-install-doc --disable-install-rdoc \ && gem update --system \ && mkdir -p "${GEM_HOME}/bin" \ - && chown -R ubuntu:ubuntu "${GEM_HOME}" + && chown -R ${USER_ID}:${GROUP_ID} "${GEM_HOME}" RUN syft_cli_path="/usr/local/bin/syft" \ && curl --show-error -sL "${SYFT_CLI_URL}" \ diff --git a/docs/apple-silicon-builds.md b/docs/apple-silicon-builds.md new file mode 100644 index 0000000000..49c2ae2959 --- /dev/null +++ b/docs/apple-silicon-builds.md @@ -0,0 +1,157 @@ +# Building on Apple Silicon + +How to build stemcells on an arm64 Mac. This is about the **build host**: the +builder image and the rake tasks are x86-64 and run under Rosetta translation. + +It is independent of the `-rosetta` stemcell *variant*, which is a property of +the stemcell you produce — see [Rosetta stemcell variant](rosetta-stemcell-variant.md). +You can build a plain `warden` stemcell here, and the variant only matters if you +intend to run the resulting stemcell under Rosetta too. + +## Prerequisites + +Colima with the `vz` VM type and Rosetta enabled (`~/.colima/default/colima.yaml`): + +```yaml +arch: aarch64 +vmType: vz +rosetta: true +mountType: virtiofs +cpu: 16 # the build is long; give it what you can spare +memory: 32 +disk: 64 +``` + +Those are the values this was verified with. The build itself is more modest: +about 6GB of scratch in `/mnt/stemcells`, a 3GB builder image, and roughly 1.2GB +of artifacts in `tmp/`. Leave headroom for the BuildKit cache, which grows +quickly across repeated attempts (`docker buildx du`, `docker buildx prune`). + +Check Rosetta is actually wired up before going further: + +```shell +docker run --rm --platform linux/amd64 ubuntu:resolute uname -m # => x86_64 +``` + +`docker buildx ls` will *not* list `linux/amd64` among the builder's platforms. +That is cosmetic: Colima's Rosetta binfmt handler is not advertised to BuildKit +but is used anyway, and `docker build --platform linux/amd64` works. + +## Build the builder image + +```shell +export short_name="resolute" + +docker build \ + --platform linux/amd64 \ + --build-arg BASE_IMAGE="ubuntu:${short_name}" \ + --build-arg USER_ID="$(id -u)" \ + --build-arg GROUP_ID="$(id -g)" \ + --build-arg ARM64_TAR_FIX=true \ + --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \ + --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \ + --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \ + --build-arg RUBY_INSTALL_URL="https://github.com/postmodern/ruby-install/releases/download/v0.10.2/ruby-install-0.10.2.tar.gz" \ + --build-arg RUBY_VERSION="$(cat .ruby-version)" \ + --build-arg GEM_HOME="/usr/local/bundle" \ + --build-arg OVF_TOOL_INSTALLER="VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle" \ + --build-arg OVF_TOOL_INSTALLER_SHA1="6c24e473be49c961cfc3bb16774b52b48e822991" \ + -t bosh/os-image-stemcell-builder:${short_name}-rosetta \ + ci/docker/os-image-stemcell-builder/ +``` + +`ARM64_TAR_FIX=true` is required here and must stay at its `false` default in +CI, where there is no arm64 emulation and the arm64 binary would not run. It +replaces the builder image's own `tar` — see +[the tar problem](#ubuntu-2604s-x86-64-tar-does-not-work-under-rosetta) below. + +Expect five to ten minutes, most of it compiling Ruby under translation. + +### ovftool + +`ovftool` is only used by the `image_ovf_generate` stage, which appears solely in +`ovf_package_stages` — vSphere and vCloud. **Warden stemcells never invoke it.** + +The `OVF_TOOL_INSTALLER` bundle is nonetheless required to build the *builder +image*, because the Dockerfile `ADD`s it unconditionally. Download it into +`ci/docker/os-image-stemcell-builder/` first. If you only ever build warden +stemcells, making that `ADD` conditional would remove the dependency. + +## Build the OS image and the stemcell + +Start a long-lived container and do both builds in it. Gems live in the +container's `/usr/local/bundle`, not in the bind mount, so a fresh `docker run` +always needs `bundle install` again; keeping one container avoids that and keeps +`/mnt/stemcells` around for re-running tests. + +```shell +export short_name="resolute" + +docker run -it --name ${short_name}-rosetta-build \ + --platform linux/amd64 \ + --privileged \ + -v "$(pwd):/opt/bosh" \ + --workdir /opt/bosh \ + --user="$(id -u):$(id -g)" \ + bosh/os-image-stemcell-builder:${short_name}-rosetta + +# You're now in the Docker container +export short_name="resolute" +gem install bundler +bundle install + +# build the OS image (the stemcell build below consumes it) +bundle exec rake stemcell:build_os_image[ubuntu,${short_name},/opt/bosh/tmp/ubuntu_base_image_${short_name}.tgz] + +# build the warden rosetta stemcell +bundle exec rake stemcell:build[warden,warden,ubuntu,${short_name}-rosetta,/opt/bosh/tmp/ubuntu_base_image_${short_name}.tgz,9.000] +``` + +Each took roughly 10-15 minutes on a 16-CPU VM with a local apt cache; expect +longer without one. Both run their RSpec suites at the end, so a clean exit +means the tests passed too. + +The OS image is architecture-neutral and carries no Rosetta-specific changes. +The `-rosetta` suffix on the *stemcell* build is what inserts the +`base_ubuntu_warden_rosetta` stage, so the same OS image tarball also serves a +plain `warden` build. + +### Rebuilding at the same version + +`create-env` records uploaded stemcells by name and version in its state file. +Rebuilding without bumping the version leaves that record stale, and the next +`create-env` reports `Skipped [Stemcell already uploaded]` and reuses the old +image. Either bump the version or drop the `stemcells` entry (and +`current_stemcell_id`) from `state.json`. + +## Speeding up repeat builds + +The build downloads well over a gigabyte of debs. If you run +[apt-cacher-ng](https://hub.docker.com/r/sameersbn/apt-cacher-ng) on the host, +add `-e http_proxy=http://host.docker.internal:3142` to `docker run` and both +debootstrap and the in-chroot `apt` calls will use it. + +## Gotchas + +### Never use a `ROSETTA_` prefix for an env var or Docker `ARG` + +Rosetta reserves that namespace and refuses to start *any* process when it sees +one it does not recognise, so the container dies instantly with `rosetta error: +invalid ROSETTA_ environment variable ...` and exit code 133. Docker turns every +`ARG` into an env var for `RUN`, which is why the tar build arg is called +`ARM64_TAR_FIX`. + +### Ubuntu 26.04's x86-64 `tar` does not work under Rosetta + +Every extraction fails with `Cannot open: Function not implemented` (ENOSYS), +because `tar` issues a syscall Rosetta does not translate. + +As of August 2026 this affects `tar` 1.35+dfsg-4ubuntu0.4 on Ubuntu 26.04. +Ubuntu 24.04's `tar` is unaffected, so it is new in 26.04, and `tar` is the only +affected tool in the base image — `gzip`, `xz`, `zstd`, `cpio`, `rsync` and +coreutils all work. `apt` also works, because `dpkg` unpacks debs with its own +built-in tar reader; only direct `tar` calls break, including `dpkg -x`. Recheck +whether the workaround is still needed when moving to a newer `tar`. + +A bare `exit code 2` from a step that unpacks an archive is almost always this. +Re-run the command by hand to see the `Function not implemented` lines. diff --git a/docs/rosetta-stemcell-variant.md b/docs/rosetta-stemcell-variant.md new file mode 100644 index 0000000000..541132310b --- /dev/null +++ b/docs/rosetta-stemcell-variant.md @@ -0,0 +1,164 @@ +# The `-rosetta` stemcell variant + +A stemcell built as `...,resolute-rosetta,...` inserts the +`base_ubuntu_warden_rosetta` stage into `warden_stages`. That stage makes the +**resulting stemcell** able to run as a container under Rosetta x86_64 emulation +on an arm64 kernel. + +This is separate from [building on Apple Silicon](apple-silicon-builds.md), +which is about the build host. A plain `warden` stemcell built on a Mac is not a +Rosetta stemcell, and this variant would be pointless on an x86-64 host. + +Only warden builds are affected; cloud infrastructure stemcells never include +the stage. Because it is gated on the variant +(`stage_collection.rb`, `operating_system.variant == "rosetta"`), the standard CI +Resolute build never exercises it — changes here have to be verified against a +deliberate `-rosetta` build. + +## Why binaries get replaced + +Rosetta does not translate every syscall, and its JIT compiler needs +writable+executable memory. A handful of x86-64 binaries therefore cannot run at +all. Each is replaced with the arm64 build of the same package version, which +runs natively on the arm64 kernel. + +arm64 libraries install alongside the x86-64 ones under `/lib/aarch64-linux-gnu/` +via Ubuntu multiarch, and the arm64 binaries find them through their built-in +RPATH. Each replaced binary is kept as `.amd64` so the swap is visible when +inspecting a stemcell. + +| binary | why the x86-64 build fails | +| --- | --- | +| `tar` | Cannot extract anything (ENOSYS). See [the tar problem](apple-silicon-builds.md#ubuntu-2604s-x86-64-tar-does-not-work-under-rosetta). Needed at runtime too: the BOSH agent shells out to `tar` for every release blob. | +| systemd daemons | systemd v256+ needs `pidfd_open` / `pidfd_send_signal`. | +| `unix_chkpwd` | AppArmor denies it the Rosetta interpreter. | +| `auditd` | Cannot satisfy `Type=forking` + `MemoryDenyWriteExecute=true`. | +| `logrotate` | Killed by `MemoryDenyWriteExecute=true` (`Result=signal`). | + +Only the systemd *daemons* in `/usr/lib/systemd/` are swapped. The CLI tools in +`/usr/bin` (`systemctl`, `journalctl`, `udevadm`) stay x86-64: they talk to PID 1 +over D-Bus and never call pidfd themselves, and keeping them x86-64 lets the +build-time RSpec suite run them inside the x86-64 chroot. + +### unix_chkpwd and AppArmor + +This one is not a missing syscall. `pam_unix` forks `/usr/sbin/unix_chkpwd` to +read `/etc/shadow`, and an AppArmor profile is attached by path to +`/{,usr/}{,s}bin/unix_chkpwd`. The profile is loaded on the Lima VM, but AppArmor +is enforced by the shared kernel, so it confines that path inside containers too. +It grants the binary and `/etc/shadow` and nothing else — in particular no access +to the Rosetta interpreter, which an x86-64 build needs in order to exec: + +```text +apparmor="DENIED" operation="open" profile="unix-chkpwd" + name="mnt/lima-rosetta/rosetta" info="Failed name lookup - disconnected path" +pam_unix(sshd:account): unix_chkpwd abnormal exit: 5 +``` + +Every PAM rule that forks the helper then fails: `su` at the auth stage, and +`bosh ssh` at the account stage, where sshd rejects the login immediately after +printing the banner even though pubkey auth succeeded. + +The interpreter cannot be allow-listed. AppArmor reports it as a *disconnected* +path — note the missing leading slash above — because it lives on a mount it +cannot resolve into the host namespace, so ordinary path rules never match. +Allowing it needs `flags=(attach_disconnected)` on the profile header, which +cannot come from an `/etc/apparmor.d/local/` include and would mean every +developer editing their own VM's vendor profile. + +Using the arm64 build removes the interpreter from the picture, so `/etc/pam.d` +can be left exactly as the packages and hardening stages produce it. + +## Before changing anything PAM-related + +`unix_chkpwd` has bitten this repo more than once, in ways that are easy to +misdiagnose. Worth reading before touching `/etc/pam.d`, `/etc/shadow` or the +helper's permissions. + +**Do not route around `unix_chkpwd` in PAM.** Replacing an account rule with +`pam_permit`, or adding `account sufficient pam_rootok.so`, resolves the symptom +by skipping the entire account stack — including the `pam_faillock` rule this repo +adds in `password_policies/assets/ubuntu/common-account.patch`, and account-expiry +enforcement. On `/etc/pam.d/su` the equivalent shortcut on the *auth* stack also +stops passwords being verified at all, letting any wheel member become root +without one. + +**`su: Authentication failure` does not mean the auth stack failed.** Ubuntu's +`/bin/su` is util-linux, not shadow, and it prints `pam_strerror()` for a failure +of *either* `pam_authenticate` or `pam_acct_mgmt`. The account stack is the more +common culprit, because that is where `pam_unix` forks `unix_chkpwd`. + +**`/etc/shadow` is mode `0400`, not the `0000` STIG V-38504 asks for.** In PAM +1.7.0, which Resolute ships, `unix_chkpwd` links `libcap-ng` and drops +`CAP_DAC_OVERRIDE` before opening `/etc/shadow`. At `0000 root:root` nothing can +read it — not even root, and not with full `CapEff` — so `unix_chkpwd … chkexpiry` +exits 9 and every `su` fails. `0400` still denies group and other, which is the +intent of the control. PAM 1.5.3 (Noble) and 1.4.0 (Jammy) are unaffected. See +`stemcell_builder/stages/base_file_permission/apply.sh`. + +**Restoring the helper's setgid bit is a red herring for that one.** +`restrict_binary_setuid` in `stemcell_builder/lib/prelude_bosh.bash` does strip it, +but `/etc/shadow` is `root:root`, so `setgid shadow` grants nothing; re-adding it +leaves `su` broken. It is also why this stage installs the arm64 helper `0755` +rather than the `2755` the deb ships — see the table above. + +**Do not reproduce shadow-permission problems on Apple Silicon.** On aarch64, +`unix_chkpwd … chkexpiry` succeeds against a `0000` shadow and `su` works, so an +aarch64 repro of that bug falsely passes. It only manifests on x86_64. (The +AppArmor problem above is the opposite: it needs Rosetta to reproduce.) + +**Do not reorder `pam_wheel` in `/etc/pam.d/su`.** `restrict_su_command` appends +`auth required pam_wheel.so use_uid` for CIS-9.5, which lands *after* the stock +file's `auth sufficient pam_rootok.so`. That ordering is what lets root bypass the +wheel check; moving the appended line up would force root through it. + +**Test against a fresh container.** Hand-edited PAM files in a long-lived debug +container will make broken configurations look like working ones: + +```shell +docker run --rm --entrypoint bash bosh.io/stemcells:img- -c '…' +``` + +## systemd-binfmt is masked + +`systemd-binfmt` rewrites `/proc/sys/fs/binfmt_misc`, which is where Lima +registers the Rosetta handler. Letting it run risks deregistering Rosetta and +leaving no x86-64 binary in the container executable. This is unrelated to the +architecture of the `systemd-binfmt` binary itself. + +## Service hardening + +The stage relaxes no systemd hardening. `MemoryDenyWriteExecute`, +`SystemCallFilter`, `LockPersonality` and `NoNewPrivileges` are left as the +vendor units ship them. + +Relaxing them is only ever correct for a service whose binary is still x86-64, +because the reason to do so is Rosetta's need for W+X memory. Before adding such +an override, check the binary's architecture: + +```shell +od -An -tu1 -j18 -N1 /usr/sbin/logrotate # 183 = arm64, 62 = x86-64 +``` + +An arm64 binary does not need it, and adding one weakens the stemcell for no +benefit. `bosh-stemcell/spec/stemcells/rosetta_spec.rb` asserts these overrides +are absent. + +## Units that cannot work in a container + +Handled in `base_warden`, not this stage, because they apply to every warden +stemcell regardless of architecture. Both are skipped with +`ConditionVirtualization=!container` rather than masked, so the stemcell still +behaves correctly if booted on a VM, and the journal records why: + +- **`audit-rules.service`** — the kernel audit subsystem is not namespaced. + `audit_netlink_ok()` in `kernel/audit.c` rejects `AUDIT_ADD_RULE`, + `AUDIT_DEL_RULE` and `AUDIT_LIST_RULES` unless the caller is in the initial PID + namespace, so `auditctl` gets `-EPERM` and the unit fails on the `-D` at line 2 + of `/etc/audit/audit.rules`. Being privileged and holding + `cap_audit_control` does not help. `/etc/audit/audit.rules` is left intact for + the STIG/CIS content checks, and `auditd` still starts because it only + `Wants=` this unit. Do not expect audit records from inside a container. +- **`netplan-configure.service`** — these images ship no `/etc/netplan`, so it + generates nothing, and its `ExecStartPost` runs `udevadm control --reload` + against the `systemd-udevd` that `base_warden` masks. diff --git a/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh b/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh index 5e714ae404..067e149042 100755 --- a/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh +++ b/stemcell_builder/stages/base_ubuntu_warden_rosetta/apply.sh @@ -1,32 +1,25 @@ #!/usr/bin/env bash -# Rosetta-specific warden stemcell stage. +# Warden stemcell stage for running under Rosetta x86_64 emulation on an arm64 +# kernel (Colima/Lima on Apple Silicon). # -# Applies two categories of fix for warden containers running under -# Lima/Rosetta x86_64 emulation on Apple Silicon (arm64 kernel): +# Rosetta does not translate every syscall, and its JIT needs writable+executable +# memory, so some x86-64 binaries cannot run. Each is replaced with the arm64 +# build of the same package version, which runs natively on the arm64 kernel: # -# 1. ARM64 SYSTEMD BINARIES -# On Apple Silicon (arm64 kernel + Rosetta x86_64 emulation), systemd v256+ -# requires pidfd_open and pidfd_send_signal syscalls that Rosetta does not -# translate for x86_64 processes, causing ENOSYS. Replacing the systemd ELF -# binaries with arm64 equivalents causes them to run natively on the arm64 -# kernel, giving full pidfd support. +# tar cannot extract anything (ENOSYS) +# systemd needs pidfd_open / pidfd_send_signal +# unix_chkpwd AppArmor denies it the Rosetta interpreter, breaking PAM +# auditd cannot satisfy Type=forking + MemoryDenyWriteExecute +# logrotate killed by MemoryDenyWriteExecute # -# arm64 shared libraries land in /lib/aarch64-linux-gnu/ via Ubuntu multiarch -# and coexist with the x86_64 libraries already in /lib/x86_64-linux-gnu/. -# The arm64 ELF binaries reference those library paths via their built-in RPATH. +# arm64 libraries install alongside the x86-64 ones under +# /lib/aarch64-linux-gnu/ via multiarch; the arm64 binaries find them by RPATH. # -# 2. ROSETTA COMPATIBILITY OVERRIDES -# Rosetta's JIT compiler requires writable+executable (W+X) memory. Several -# systemd services have security hardening that blocks W+X (MemoryDenyWriteExecute, -# SystemCallFilter, etc.), causing them to crash on startup. We apply drop-in -# overrides to disable these restrictions. AppArmor also blocks unix-chkpwd -# from accessing the Rosetta runtime path, causing su to fail; we replace -# /etc/pam.d/su with a config that avoids unix-chkpwd entirely. +# See docs/rosetta-stemcell-variant.md for the detail behind each entry. # -# This stage is only inserted into warden_stages when the OS variant is -# "rosetta" (i.e. rake ... ubuntu,resolute-rosetta,...). Standard warden -# and all cloud infrastructure stemcell builds are unaffected. +# Inserted into warden_stages only when the OS variant is "rosetta" +# (rake ... ubuntu,resolute-rosetta,...); other builds are unaffected. set -e @@ -34,18 +27,21 @@ base_dir=$(readlink -nf $(dirname $0)/../..) source $base_dir/lib/prelude_apply.bash # --------------------------------------------------------------------------- -# Part 1: arm64 systemd binaries +# Part 1: arm64 foreign architecture and runtime libraries # --------------------------------------------------------------------------- # Enable arm64 as a foreign architecture so apt can resolve arm64 packages. run_in_chroot $chroot "dpkg --add-architecture arm64" run_in_chroot $chroot "apt-get update" -# Install arm64 runtime libraries that systemd depends on. +# Install arm64 runtime libraries needed by the arm64 binaries installed below +# (systemd in Part 3; unix_chkpwd, auditd and logrotate in Part 4 — auditd needs +# libauparse, logrotate needs libpopt). # These install into /lib/aarch64-linux-gnu/ and /usr/lib/aarch64-linux-gnu/, # coexisting safely with the existing amd64 libraries. # libc6:arm64 also places /lib/ld-linux-aarch64.so.1 (the arm64 ELF interpreter). arm64_libs="libc6:arm64 \ + libselinux1:arm64 \ libcrypt1:arm64 \ libgcrypt20:arm64 \ libblkid1:arm64 \ @@ -61,10 +57,75 @@ arm64_libs="libc6:arm64 \ libssl3t64:arm64 \ libp11-kit0:arm64 \ liblzma5:arm64 \ - libgpg-error0:arm64" + libgpg-error0:arm64 \ + libauparse0t64:arm64 \ + libpopt0:arm64" run_in_chroot $chroot "apt-get install --no-install-recommends --assume-yes $arm64_libs" +# --------------------------------------------------------------------------- +# Part 2: arm64 tar +# --------------------------------------------------------------------------- + +# This has to happen before the systemd debs below, because extracting those +# uses the chroot's own tar — and Ubuntu 26.04's x86-64 GNU tar +# (1.35+dfsg-4ubuntu0.4) cannot extract anything under Rosetta: every file it +# creates fails with "Cannot open: Function not implemented" (ENOSYS), because +# tar issues a syscall Rosetta does not translate. Ubuntu 24.04's tar is +# unaffected, so this is new in 26.04. The arm64 build of the same version works +# because it runs natively on the arm64 kernel. It is also the only tool in the +# base image with this problem — gzip, xz, zstd, cpio, rsync and coreutils are +# all fine. +# +# It matters at runtime too, not just during the build: the BOSH agent shells +# out to tar to unpack every compiled package and release blob it downloads, so +# a stemcell with a broken tar cannot run a single deployment. +# +# dpkg has its own built-in tar reader and does not call GNU tar to unpack debs, +# so apt keeps working regardless; only direct tar invocations are affected +# (including `dpkg -x`, which does shell out to GNU tar). +# +# `apt-get install tar:arm64` cannot be used: tar is Multi-Arch-incompatible and +# tar:amd64 declares Conflicts: tar:arm64. Download and swap the binary in by +# hand instead, the same way the systemd binaries are handled below. +run_in_chroot $chroot " + mkdir -p /tmp/arm64-tar + cd /tmp/arm64-tar + apt-get download tar:arm64 +" + +# Unpack from outside the chroot: the chroot's own tar is the broken binary we +# are about to replace, so it cannot be used to extract its own replacement. +# The builder container's tar is known-good (see ARM64_TAR_FIX in +# ci/docker/os-image-stemcell-builder/Dockerfile). +mkdir -p "$chroot/tmp/arm64-tar/root" +dpkg-deb --fsys-tarfile "$chroot"/tmp/arm64-tar/tar_*_arm64.deb \ + | tar -x -C "$chroot/tmp/arm64-tar/root" + +# Keep the x86-64 binary as tar.amd64 for debugging and so the swap is obvious +# to anyone inspecting the stemcell. +mv "$chroot/usr/bin/tar" "$chroot/usr/bin/tar.amd64" +install -m 0755 "$chroot/tmp/arm64-tar/root/usr/bin/tar" "$chroot/usr/bin/tar" +rm -rf "$chroot/tmp/arm64-tar" + +# Fail loudly if the replacement cannot actually extract. Everything below +# depends on it, and a silently broken tar would otherwise only surface much +# later as an unexplained deployment failure. +run_in_chroot $chroot " + set -e + rm -rf /tmp/tar-selftest + mkdir -p /tmp/tar-selftest/src /tmp/tar-selftest/out + echo rosetta > /tmp/tar-selftest/src/probe + tar -czf /tmp/tar-selftest/probe.tgz -C /tmp/tar-selftest src + tar -xzf /tmp/tar-selftest/probe.tgz -C /tmp/tar-selftest/out + grep -q rosetta /tmp/tar-selftest/out/src/probe + rm -rf /tmp/tar-selftest +" + +# --------------------------------------------------------------------------- +# Part 3: arm64 systemd binaries +# --------------------------------------------------------------------------- + # Download arm64 systemd debs without installing. # `apt-get install systemd:arm64` would conflict with systemd:amd64 because # systemd does not carry Multi-Arch: same. Download and extract manually instead. @@ -114,86 +175,84 @@ run_in_chroot $chroot " run_in_chroot $chroot "rm -rf /tmp/arm64-debs" # --------------------------------------------------------------------------- -# Part 2: Rosetta compatibility overrides +# Part 4: arm64 userland binaries # --------------------------------------------------------------------------- -# Apply systemd drop-ins that disable security hardening features conflicting -# with Rosetta's JIT compilation (which requires writable+executable memory). -rosetta_services=( - systemd-journald - systemd-resolved - systemd-networkd - systemd-logind - systemd-timesyncd - systemd-udevd - logrotate - auditd -) - -for service in "${rosetta_services[@]}"; do - mkdir -p "$chroot/etc/systemd/system/${service}.service.d" - cp "$assets_dir/rosetta-compat.conf" "$chroot/etc/systemd/system/${service}.service.d/rosetta-compat.conf" -done - -# Mask systemd-binfmt.service which fails under Rosetta emulation. -run_in_chroot "$chroot" "systemctl mask systemd-binfmt.service" - -# auditd: systemd can return ENOSYS when creating pidfd/cgroup references from -# PIDFile under Docker/Colima. Foreground auditd avoids forking + PIDFile quirks. -mkdir -p "$chroot/etc/systemd/system/auditd.service.d" -cat > "$chroot/etc/systemd/system/auditd.service.d/warden-auditd-foreground.conf" <<'UNIT' -[Service] -Type=simple -PIDFile= -ExecStart= -ExecStart=/usr/sbin/auditd -n -UNIT - -# Ubuntu enables OpenSSH via ssh.socket (socket activation). systemd's listener -# stub fork can fail with ENOSYS under Docker/Colima with Rosetta x86_64 -# emulation. Use the traditional ssh.service so sshd binds port 22 itself. -mkdir -p "$chroot/etc/systemd/system/ssh.service.d" -cat > "$chroot/etc/systemd/system/ssh.service.d/warden-no-socket-activation.conf" <<'UNIT' -[Unit] -# When ssh.socket is masked, sshd must start via ssh.service; drop RefuseManualStart from the vendor unit. -RefuseManualStart=no -UNIT -run_in_chroot "$chroot" "systemctl mask ssh.socket" -run_in_chroot "$chroot" "systemctl enable ssh.service" - -# Fix `su` PAM authentication failures under Colima/Lima (Apple Silicon). +# swap_arm64_binary [group] # -# Under Lima's Rosetta x86_64 emulation, AppArmor blocks unix-chkpwd -# (an x86_64 binary) from accessing the Rosetta runtime path -# /mnt/lima-rosetta/rosetta, causing every `su` invocation to fail with -# "Authentication failure" — even for root. The standard PAM config in -# /etc/pam.d/su includes common-auth, which chains pam_faillock → pam_unix, -# and pam_unix forks unix-chkpwd to verify passwords. That fork triggers -# the AppArmor denial. +# None of these packages carry Multi-Arch: same, so `apt-get install :arm64` +# would conflict with the amd64 package; download and swap by hand instead. The +# x86-64 build is kept as .amd64, matching tar.amd64 above. +swap_arm64_binary() { + local pkg="$1" path="$2" group="${3:-}" + local stage="/tmp/arm64-swap" + + run_in_chroot $chroot " + set -e + rm -rf $stage + mkdir -p $stage + cd $stage + apt-get download ${pkg}:arm64 + dpkg -x $stage/${pkg}_*_arm64.deb $stage/root + " + + if [ ! -f "$chroot$stage/root$path" ]; then + echo "ERROR: ${pkg}:arm64 does not ship $path" >&2 + exit 1 + fi + + mv "$chroot$path" "$chroot${path}.amd64" + install -m 0755 "$chroot$stage/root$path" "$chroot$path" + [ -n "$group" ] && run_in_chroot $chroot "chown root:$group $path" + run_in_chroot $chroot "rm -rf $stage" + + # Fail loudly if the swap did not land. A silently-still-x86-64 binary would + # only resurface much later as an unexplained runtime failure. + run_in_chroot $chroot " + set -e + head -c 20 $path | grep -qa 'ELF' || { echo '$path is not an ELF binary' >&2; exit 1; } + # e_machine at offset 18 is 183 (EM_AARCH64) for arm64, 62 for x86-64. + arch_byte=\$(od -An -tu1 -j18 -N1 $path | tr -d ' ') + if [ \"\$arch_byte\" != '183' ]; then + echo \"ERROR: $path is not arm64 (e_machine=\$arch_byte)\" >&2 + exit 1 + fi + " +} + +# pam_unix forks /usr/sbin/unix_chkpwd to read /etc/shadow. An AppArmor profile +# attached by path to /{,usr/}{,s}bin/unix_chkpwd — loaded on the Lima VM and +# enforced by the shared kernel, so it applies inside containers too — grants no +# access to the Rosetta interpreter, which an x86-64 build needs in order to exec: # -# pam_rootok.so already handles "root switching to any user" correctly on -# its own, but because pam_unix / pam_faillock come after it in common-auth -# they still run (pam_rootok is "sufficient" only when it *succeeds* at the -# auth step level, not at the include level). Using pam_permit for the -# remaining auth/account rules means root can always su without unix-chkpwd. +# apparmor="DENIED" profile="unix-chkpwd" name="mnt/lima-rosetta/rosetta" +# pam_unix(sshd:account): unix_chkpwd abnormal exit: 5 # -# pam_wheel.so use_uid is retained (CIS-9.5): it only checks group membership -# and never forks unix-chkpwd, so it is safe under Rosetta/AppArmor. -# pam_rootok is "sufficient", so root bypasses the wheel check entirely; -# non-root users must be in the wheel group (vcap is added by restrict_su_command). +# The interpreter cannot be allow-listed: AppArmor reports it as a disconnected +# path (note the missing leading slash), which needs flags=(attach_disconnected) +# on the profile header and so cannot come from an /etc/apparmor.d/local/ include. # -# This override is safe for warden containers: the host provides isolation, -# and the container root user is already fully privileged. -cat > "$chroot/etc/pam.d/su" <<'PAMEOF' -# PAM configuration for su - warden Rosetta stemcell override. -# AppArmor blocks unix-chkpwd under Lima/Rosetta causing su to fail even for root. -# pam_rootok handles legitimate root → any-user su; pam_permit covers the rest. -# pam_wheel.so use_uid is kept for CIS-9.5 compliance (does not invoke unix-chkpwd). -auth sufficient pam_rootok.so -auth required pam_wheel.so use_uid -auth required pam_permit.so -account required pam_permit.so -session required pam_env.so readenv=1 -session required pam_limits.so -PAMEOF -chmod 0644 "$chroot/etc/pam.d/su" +# Mode stays 0755, not the 2755 the deb ships: restrict_binary_setuid strips +# setgid outside the allowlist asserted in stemcells/ubuntu_spec.rb. su is +# setuid-root and sshd is already privileged when they fork the helper, which is +# how it reads /etc/shadow at 0400 root:root. +swap_arm64_binary libpam-modules-bin /usr/sbin/unix_chkpwd shadow + +# auditd.service is Type=forking with a PIDFile and MemoryDenyWriteExecute=true; +# an x86-64 build can satisfy neither under Rosetta. Only the daemon is swapped: +# auditctl and augenrules cannot work in a container on any architecture, which +# is why base_warden skips audit-rules.service. +swap_arm64_binary auditd /usr/sbin/auditd + +# logrotate.service ships MemoryDenyWriteExecute=true, which kills the x86-64 +# binary outright (Result=signal). +swap_arm64_binary logrotate /usr/sbin/logrotate + +# --------------------------------------------------------------------------- +# Part 5: binfmt_misc +# --------------------------------------------------------------------------- + +# systemd-binfmt rewrites /proc/sys/fs/binfmt_misc, where Lima registers the +# Rosetta handler; letting it run risks deregistering Rosetta and leaving no +# x86-64 binary in the container executable. +run_in_chroot "$chroot" "systemctl mask systemd-binfmt.service" diff --git a/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf b/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf deleted file mode 100644 index 0f1912bd2b..0000000000 --- a/stemcell_builder/stages/base_ubuntu_warden_rosetta/assets/rosetta-compat.conf +++ /dev/null @@ -1,12 +0,0 @@ -[Service] -# Disable security features that conflict with Rosetta x86_64 emulation -# on Apple Silicon Macs. Rosetta uses JIT compilation which requires -# writable+executable memory. -# -# These overrides are safe for warden stemcells since they run in -# containerized environments where the host provides security isolation. -MemoryDenyWriteExecute=no -SystemCallArchitectures= -SystemCallFilter= -LockPersonality=no -NoNewPrivileges=no diff --git a/stemcell_builder/stages/base_warden/apply.sh b/stemcell_builder/stages/base_warden/apply.sh index ce367daded..abeb71484d 100755 --- a/stemcell_builder/stages/base_warden/apply.sh +++ b/stemcell_builder/stages/base_warden/apply.sh @@ -66,3 +66,24 @@ run_in_chroot "$chroot" "systemctl mask nvmf-autoconnect.service" # systemd-udevd has no function in containerised environments. run_in_chroot "$chroot" "systemctl mask systemd-udevd.service" + +# Audit rule changes need the initial PID namespace (audit_netlink_ok() in +# kernel/audit.c returns -EPERM otherwise), so audit-rules.service can never +# succeed in a container and would sit permanently failed. Skipped by condition +# rather than masked so the stemcell still loads the rules if booted on a VM. +# /etc/audit/audit.rules is left intact for the STIG/CIS content checks, and +# auditd only Wants= this unit so it still starts. +mkdir -p "$chroot/etc/systemd/system/audit-rules.service.d" +cat > "$chroot/etc/systemd/system/audit-rules.service.d/warden-skip-in-container.conf" <<'UNIT' +[Unit] +ConditionVirtualization=!container +UNIT + +# These images ship no /etc/netplan, so netplan-configure has nothing to +# generate, and its ExecStartPost runs `udevadm control --reload` against the +# systemd-udevd masked above. +mkdir -p "$chroot/etc/systemd/system/netplan-configure.service.d" +cat > "$chroot/etc/systemd/system/netplan-configure.service.d/warden-skip-in-container.conf" <<'UNIT' +[Unit] +ConditionVirtualization=!container +UNIT diff --git a/stemcell_builder/stages/bosh_monit/apply.sh b/stemcell_builder/stages/bosh_monit/apply.sh index e8ed76903c..fde7b1d9c6 100755 --- a/stemcell_builder/stages/bosh_monit/apply.sh +++ b/stemcell_builder/stages/bosh_monit/apply.sh @@ -10,14 +10,18 @@ monit_basename=monit-5.2.5 monit_archive=$monit_basename.tar.gz mkdir -p $chroot/$bosh_dir/src -cp -r $dir/assets/$monit_archive $chroot/$bosh_dir/src + +# Unpack outside the chroot rather than with `run_in_bosh_chroot ... tar zxvf`. +# The chroot's own tar is the target OS's binary, and Ubuntu 26.04's x86-64 tar +# cannot extract under Rosetta (ENOSYS, see the ARM64_TAR_FIX note in +# ci/docker/os-image-stemcell-builder/Dockerfile). The builder container's tar +# is known-good, and unpacking a source tarball needs no chroot context. +tar zxf $dir/assets/$monit_archive -C $chroot/$bosh_dir/src pkg_mgr install "zlib1g-dev libcrypt-dev" run_in_bosh_chroot $chroot " -cd src -tar zxvf $monit_archive -cd $monit_basename +cd src/$monit_basename ./configure --prefix=$bosh_dir --without-ssl CFLAGS='-fcommon' LIBS='-lcrypt' make -j4 && make install "