Skip to content

fix(tmachine): provide a CA bundle for host tools on macOS - #4930

Closed
matthewgrossman wants to merge 1 commit into
mainfrom
fix/tmachine-macos-ca-bundle/matthewgrossman
Closed

matthewgrossman wants to merge 1 commit into
mainfrom
fix/tmachine-macos-ca-bundle/matthewgrossman

Conversation

@matthewgrossman

Copy link
Copy Markdown
Member

Summary

On macOS, nix run .#tmachine can fail before a VM boots: ansible-galaxy can't verify galaxy.ansible.com (CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate). The tmachine wrapper now falls back to the pinned nixpkgs CA bundle when the user hasn't configured one.

Related Issue

No separate issue. This fixes a broken build/test-tooling path on macOS. Root cause and verification are below.

Root cause

Nix's OpenSSL on Darwin is compiled to trust only /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt. Python's ssl.get_default_verify_paths() in the wrapper's Ansible (Python 3.14, OpenSSL 3.6.3) reports exactly that path, with no capath fallback. The official Nix installer creates that profile and exports NIX_SSL_CERT_FILE. Other installs, including flox-managed Nix, don't, so every HTTPS client linked against Nix's OpenSSL has no trust store. SSL_CERT_DIR=/etc/ssl/certs doesn't help because macOS doesn't keep hashed certificates there.

This is not specific to ansible-galaxy or to how tmachine builds its environment. A plain urllib request from the same Python fails the same way, and system curl and /usr/bin/python3 succeed. On Linux, Nix's OpenSSL reads /etc/ssl/certs/ca-certificates.crt, which is why CI is unaffected.

Only ansible-galaxy uses TLS on the host. The guest runs Ansible modules with /usr/bin/python3 against Ubuntu's own trust store, and base images are fetched by Nix at build time.

Changes

  • tests/tmachine/config.nix: add a shared hostCaBundleSetup snippet, used by the tmachine wrapper. On Darwin, if neither NIX_SSL_CERT_FILE nor SSL_CERT_FILE is set, it exports SSL_CERT_FILE to ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt. On other platforms it is empty, and the Linux wrapper contains no CA setting.

Design notes:

  • Conditional, not runtimeEnv. runtimeEnv exports unconditionally. Nix's OpenSSL gives NIX_SSL_CERT_FILE precedence over SSL_CERT_FILE, so setting either one unconditionally could override a corporate CA bundle a user configured on purpose. The wrapper checks both and exports only SSL_CERT_FILE, the lower-precedence variable.
  • pkgs.cacert rather than /etc/ssl/cert.pem. It's pinned with the rest of the toolchain and matches what the official Nix installer provides. /etc/ssl/cert.pem doesn't include CAs from the macOS Keychain either, so it adds nothing for corporate setups.
  • Darwin only. Linux already works through the system bundle, which may include locally added CAs. Applying the fallback there would replace that bundle on hosts that don't set either variable.
  • feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773: the openshell-test-runner wrapper it adds also runs Ansible, and it lives next to this one in config.nix. It should include ${hostCaBundleSetup} in its text as well. I can follow up on whichever PR lands second.

Testing

All on Apple Silicon macOS with SSL_CERT_FILE, NIX_SSL_CERT_FILE and SSL_CERT_DIR unset:

  • Reproduced the bug: the wrapper's Python fails urllib to https://galaxy.ansible.com/api/, and ansible-galaxy role install through the old wrapper's environment fails with the reported error.

  • With the new wrapper's environment, ansible-galaxy role install succeeds.

  • User settings still win: through the new wrapper, an empty SSL_CERT_FILE or NIX_SSL_CERT_FILE still makes Galaxy fail (the user value is respected), and SSL_CERT_FILE=/etc/ssl/cert.pem succeeds.

  • End to end: tmachine install ubuntu-docker-rootful deb with an empty tmachine cache, so Galaxy and both layers run. The roles install, the VM boots, the nextest, Docker and OpenShell .deb playbooks pass, and the command exits 0.

  • nix fmt shows no changes. The x86_64-linux tmachine derivation evaluates and contains no CA setting.

  • Not run: Linux end to end (no behavior change there) and a test suite run. The local artifacts don't include test archives.

  • Checks appropriate to the affected code and behavior pass

  • Unit tests added/updated (if applicable)

  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Nix's OpenSSL on Darwin only trusts the CA bundle in the default Nix
profile. Installers that don't create that profile leave ansible-galaxy
unable to verify galaxy.ansible.com, so environments with use_galaxy fail
before the VM boots.

When neither NIX_SSL_CERT_FILE nor SSL_CERT_FILE is set, the tmachine
wrapper now points SSL_CERT_FILE at the pinned nixpkgs cacert bundle.
User-configured bundles keep precedence.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

Closing without merging: the root cause was how Nix was installed on my machine, not OpenShell.

What was wrong. Nix's OpenSSL on Darwin trusts only /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt unless NIX_SSL_CERT_FILE or SSL_CERT_FILE is set. Nix had come in as a dependency of the Flox Homebrew cask, and that install created neither the default profile nor the shell variable. It set NIX_SSL_CERT_FILE only for the daemon. So every client-side process using Nix's OpenSSL had no trust store: tmachine's ansible-galaxy, and also the nix CLI's own flake fetches (nix fmt failed fetching treefmt-nix). The official and Determinate installers both set this up, which is why CI and most machines never hit it.

What we changed locally.

  • Removed Flox and its Nix with brew uninstall --zap flox. The first attempt failed because Microsoft Defender's DLP agent blocked unmounting the Nix store volume. A retry succeeded.
  • Cleaned up the uninstaller's leftovers: a stale /nix line in /etc/fstab (its cleanup pattern didn't match), /etc/nix, and a few files Homebrew couldn't remove. Then rebooted.
  • Installed Determinate Nix (3.24.1, Nix 2.35.3), the same distribution CI uses through .github/actions/setup-nix. It creates the default-profile CA bundle, sets NIX_SSL_CERT_FILE in /etc/zshrc, and sets Nix's own ssl-cert-file to a copy of the macOS Keychain certificates.

Verified after the reinstall, using main's wrapper without this change and with SSL_CERT_FILE, NIX_SSL_CERT_FILE and SSL_CERT_DIR unset:

  • ansible-galaxy installs geerlingguy.docker.
  • nix fmt works.
  • tmachine install ubuntu-docker-rootful deb passes with an empty cache: Galaxy, VM boot, nextest, Docker and the .deb install (exit 0).

The fallback in this PR would only help Macs whose Nix install is missing the default profile. That doesn't seem worth carrying in the wrapper or adding to #4773's runner. If someone else hits CERTIFICATE_VERIFY_FAILED from tmachine on macOS, check whether /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt exists. Reinstalling Nix with the Determinate installer fixes it. Setting NIX_SSL_CERT_FILE works as a stopgap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants