Skip to content

WIP: [ruby] Resolve @rpath/@loader_path dylib references when relinking - #5160

Open
okorohelijah wants to merge 6 commits into
flutter:mainfrom
okorohelijah:fix/ruby-relink-rpath
Open

okorohelijah wants to merge 6 commits into
flutter:mainfrom
okorohelijah:fix/ruby-relink-rpath

Conversation

@okorohelijah

@okorohelijah okorohelijah commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What failed

Mac_arm64 ruby has been failing since the Xcode bump in #5155 let it get past Homebrew and actually compile Ruby. It then died in the vendored packaging step, cipd_packages/ruby/third_party/ruby_ship/auto_relink_dylibs.rb:

fileutils.rb:1394:in 'initialize': No such file or directory @ rb_sysopen - @rpath/libbrotlicommon.1.dylib (Errno::ENOENT)
  from .../tools/auto_relink_dylibs.rb:40:in 'block in fix_dylib_for_file'

Failing prod build: https://ci.chromium.org/b/8669644619614462833

While verifying the fix on led, three more, unrelated breakages showed up behind it, one after another. This PR fixes all four.

1. @rpath dependencies in auto_relink_dylibs.rb

The script walks otool -L for every file in bin/darwin_ruby, copies each dependency into bin/darwin_ruby/dylibs/ using the literal path otool printed, and then recurses into the copy. Homebrew's brotli 1.2.0 bottle (a curl dependency) references its sibling as @rpath/libbrotlicommon.1.dylib. That is a dyld search token, not a filesystem path, so FileUtils.copy blows up. And because the script recurses into the copied libbrotlidec.1.dylib, it has also lost track of where the original lived.

Fix:

  • Thread the original source directory through the recursion.
  • When a dependency starts with @rpath/, @loader_path/ or @executable_path/, resolve it to a real file before copying, trying in order: the referencing library's original directory, each LC_RPATH from otool -l (with @loader_path expanded against the original directory), and finally the dylibs/ directory in case it was already copied. @executable_path depends on whichever executable loads the library, which isn't known at packaging time, so those are matched by basename only.
  • If none of those exist, print which reference could not be resolved and exit 1, rather than failing with an opaque ENOENT.
  • install_name_tool -change still gets the exact original string from the load command as old, so the rewrite matches.

Absolute paths behave exactly as before; /usr/lib/ and /System/Library/ are still skipped.

2. brew --prefix openssl now points at openssl@4

On 2026-09-27 homebrew-core moved the openssl alias from openssl@3 to openssl@4 (Homebrew/homebrew-core@9a12b94a). build.sh installs openssl@3, but ruby_ship_build.sh resolved brew --prefix openssl, which now returns /opt/homebrew/opt/openssl@4, a directory that does not exist on the bots. The bogus -L/opt/homebrew/opt/openssl@4/lib makes ld warn, Ruby 3.1's configure treats warnings under -Werror as failed probes (checking for function name string predefined identifier... no), and make dies with use of undeclared identifier 'RUBY_FUNCTION_NAME_STRING'. The last prod run (two days earlier) still resolved the alias to openssl@3, so prod would hit this on its next run regardless of the relink fix. Pinned to brew --prefix openssl@3 in the two places it is used.

3. gem install -f bundler picks up bundler 4

bundler 4.0.21 (released 2026-09-16) requires Ruby >= 3.2 and RubyGems >= 3.4.1. -f bypasses that check, so it was force-installed onto this Ruby 3.1.3 build and the bin/bundler --version smoke test crashed inside the RubyGems resolver (rubygems/resolver/conflict.rb:47: undefined method 'request' for nil:NilClass). Pinned to -v '~> 2.5.0', which resolves to 2.5.23 (Ruby >= 3.0, RubyGems >= 3.2.3). A bare ~> 2.5 would resolve to 2.7.2, which already needs Ruby >= 3.2, hence the patch-level constraint.

4. pod --version fails to load ActiveSupport (flutter/flutter#162341)

With everything above fixed, the last smoke test failed with the error this builder used to produce back in mid-2025 whenever it got this far:

activesupport-7.0.8/lib/active_support/logger_thread_safe_level.rb:12: uninitialized constant ActiveSupport::LoggerThreadSafeLevel::Logger (NameError)

activesupport 7.0.8 uses Logger without requiring it and only worked because concurrent-ruby <= 1.3.4 happened to require "logger"; concurrent-ruby 1.3.5 (2025-01-15) removed that, and gem install activesupport -v 7.0.8 pulls the newest concurrent-ruby (1.3.8). Two commits deal with it:

  • Install concurrent-ruby 1.3.4 before activesupport. Every consumer in the package accepts it (activesupport ~> 1.0, >= 1.0.2, cocoapods-core ~> 1.1, i18n and tzinfo ~> 1.0).
  • That alone turned out not to be enough on the RubyGems that ships with Ruby 3.1.3 (3.3.26): it still installs the newest concurrent-ruby next to 1.3.4, and pod activates the newest one. Newer RubyGems keeps the installed 1.3.4, which is why a local dry run looked fine. So after the cocoapods install the script now runs gem uninstall concurrent-ruby -a -x -I -v '> 1.3.4', which removes anything newer and is a no-op (exit 0) when there is nothing to remove.

Note: third_party/ruby_ship is vendored from stephan-nordnes-eriksen/ruby_ship. ruby_ship_build.sh was already modified locally in #3186; this is the first local change to auto_relink_dylibs.rb.

How it was verified

  • bash -n on the shell script; ruby -c, plus a throwaway minitest (not committed) covering absolute paths, @rpath next to the original, @rpath through absolute and @loader_path LC_RPATH entries, @loader_path/... relative references, @executable_path basename fallback (and that its relative form is not expanded), the already-copied fallback, the unresolvable case, and otool -l parsing including CRLF/tab output. 12 tests, all green.
  • An end-to-end run of the script on Linux with fake otool/install_name_tool reproducing the brotli layout (libcurl -> /abs/libbrotlidec.1.dylib -> @rpath/libbrotlicommon.1.dylib): all three libraries land in dylibs/, -change uses @rpath/libbrotlicommon.1.dylib as the old name, and removing libbrotlicommon makes it exit 1 with a clear message.
  • led runs on the staging pool (Mac_arm64 ruby, checking out refs/pull/5160/head; CIPD upload is skipped on led because the recipe only registers from the prod bucket):
    • Run 1 (relink fix only): https://ci.chromium.org/b/8669384638160948033 — failed before the relink step with the openssl@4 problem in (2).
    • Run 2 (relink + openssl@3): https://ci.chromium.org/b/8669383997159272673 — Ruby configures and compiles again; all three passes of auto_relink_dylibs.rb finish with Relinking of dylib done, and the previously failing spot now reads --COPIED-- .../dylibs/libbrotlicommon.1.dylib / Linked: .../libbrotlicommon.1.dylib to .../libbrotlidec.1.dylib. bundler, activesupport 7.0.8 and cocoapods 1.16.2 install; then bin/bundler --version crashed with the bundler 4 problem in (3).
    • Run 3 (all three fixes): https://ci.chromium.org/b/8669376654066365665 — landed on flutter-devicelab-mac-5, which has no Command Line Tools, so Homebrew stopped at brew reinstall --build-from-source gdbm with Error: Xcode alone is not sufficient on Sequoia (Mac ruby failing due to missing command line tools flutter#162344). Nothing to do with this change; the staging arm64 fleet is just inconsistent on CLT.
    • Run 3b (same commits, pinned to flutter-devicelab-mac-55): https://ci.chromium.org/b/8669375943591764321 — configure/compile fine, all three relink passes finish, bundler 2.5.23 / activesupport 7.0.8 / cocoapods 1.16.2 install; bin/bundler --version prints Bundler version 2.5.23, bin/gem --version prints 3.3.26; fails at bin/pod --version with the ActiveSupport Logger NameError in (4).
    • Run 4 (with the concurrent-ruby 1.3.4 pre-install): https://ci.chromium.org/b/8669366981838817409 — log shows Successfully installed concurrent-ruby-1.3.4 and then, during the activesupport install, Successfully installed concurrent-ruby-1.3.8; pod --version fails the same way. That is what prompted the uninstall step.
    • Run 5 (all six commits): https://ci.chromium.org/b/8669365556185906913 — green. Build status SUCCESS in 21 minutes. ruby_ship_build.sh reaches its DONE banner; the log shows Successfully installed concurrent-ruby-1.3.4, later Successfully installed concurrent-ruby-1.3.8 (pulled in by activesupport, as in run 4), then Successfully uninstalled concurrent-ruby-1.3.8 from the new step. All four smoke tests in tools/build.sh pass: bin/bundle --version → ri 6.4.0 (see note below), bin/bundler --version → Bundler version 2.5.23, bin/gem --version → 3.3.26, bin/pod --version → 1.16.2. The recipe's build mac-arm64 step then packaged build/ into a CIPD instance (flutter/ruby/mac-arm64, instance qTUDFbJFw8l9dWrWncVXCope1qBH-Rc1JUr8UGnQ6qMC). Nothing was uploaded: cipd.py L72–73 only registers when the bucket is prod and the build is for refs/heads/main; a led build runs in staging.shadow with no gitiles commit, so the register step is skipped by design.

Note: bin/bundle --version printing ri 6.4.0 is a pre-existing bug in the vendored script — ruby_ship_build.sh L126–129 generates bin/bundle pointing at the ri wrapper instead of darwin_bundle.sh. It exits 0 so it never failed the build; left for a separate one-line follow-up.

Out of scope

Mac ruby (Intel) is separately blocked on missing Command Line Tools (flutter/flutter#162344) and is not touched here.

Related: flutter/flutter#164665

The `Mac_arm64 ruby` CIPD builder fails in the vendored ruby_ship
packaging step (auto_relink_dylibs.rb) after Ruby has been compiled:

  fileutils.rb:1394:in 'initialize': No such file or directory @
  rb_sysopen - @rpath/libbrotlicommon.1.dylib (Errno::ENOENT)
  from auto_relink_dylibs.rb:40 (FileUtils.copy)

The script walks `otool -L` output and copies every dependency into
bin/darwin_ruby/dylibs using the literal string reported by otool.
Homebrew's brotli 1.2.0 bottle (pulled in by curl) references its
sibling library as `@rpath/libbrotlicommon.1.dylib`, which is a dyld
search token rather than a filesystem path, so the copy fails. Because
the script recurses into the *copied* dylib, it also no longer knows
where the original lived and could not find the sibling even if it
tried.

Fix:
- Thread the original source directory through the recursion.
- Before copying an `@rpath/`, `@loader_path/` or `@executable_path/`
  dependency, resolve it to a real file by trying, in order: the
  referencing library's original directory, each LC_RPATH entry from
  `otool -l` (with `@loader_path` expanded against the original
  directory), and finally the dylibs directory itself if the library
  was already copied.
- If none of those exist, print which reference could not be resolved
  and exit 1 instead of crashing with an opaque ENOENT.
- `install_name_tool -change` still receives the exact original string
  from the load command as its `old` argument, so the rewrite matches.

Absolute paths behave exactly as before, and `/usr/lib/` and
`/System/Library/` entries are still skipped. Note that this script is
vendored from stephan-nordnes-eriksen/ruby_ship (third_party/ruby_ship);
this is the first local change to the .rb file (ruby_ship_build.sh was
already modified locally in flutter#3186).

Failing build: https://ci.chromium.org/b/8669644619614462833

Related: flutter/flutter#164665

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for resolving relative Mach-O dependencies (such as @rpath, @loader_path, and @executable_path) in auto_relink_dylibs.rb by parsing otool -l output and searching for candidates. The review feedback highlights three key areas for improvement: correctly resolving @executable_path relative to the main executable directory instead of the referring library's directory, making the otool -l output regex parser more robust against whitespace variations, and escaping file paths passed to shell commands to prevent shell injection or execution failures.

Comment on lines +31 to +47
def resolve_dylib_path(libfile, source_dir, rpaths)
return libfile unless libfile.start_with?("@")

basename = File.split(libfile)[-1]
candidates = []
if libfile.start_with?("@loader_path/", "@executable_path/")
candidates << File.expand_path(libfile.sub(/\A@(loader_path|executable_path)/, source_dir))
end
candidates << File.join(source_dir, basename)
rpaths.each do |rpath|
rpath = File.expand_path(rpath.sub(/\A@(loader_path|executable_path)/, source_dir))
candidates << File.join(rpath, basename)
end
candidates << File.join(@new_dylib_path, basename)

candidates.find { |candidate| File.file?(candidate) }
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Currently, @executable_path is resolved to source_dir (the directory of the referring library). However, @executable_path refers to the directory of the main executable (e.g., bin/darwin_ruby/bin/), whereas @loader_path refers to the directory of the referring library. If a library references a dependency via @executable_path, resolving it to source_dir will fail to find the dependency if the referring library is located in dylibs/ or lib/.

We can resolve @executable_path correctly by computing the executable directory relative to @new_dylib_path (which is File.expand_path(File.join(@new_dylib_path, "..", "bin"))).

Additionally, we can simplify the substitution logic and use the more idiomatic File.basename instead of File.split(libfile)[-1].

def resolve_dylib_path(libfile, source_dir, rpaths)
	return libfile unless libfile.start_with?("@")

	basename = File.basename(libfile)
	executable_dir = File.expand_path(File.join(@new_dylib_path, "..", "bin"))
	candidates = []
	if libfile.start_with?("@loader_path/", "@executable_path/")
		resolved_libfile = libfile.sub(/\A@loader_path/, source_dir).sub(/\A@executable_path/, executable_dir)
		candidates << File.expand_path(resolved_libfile)
	end
	candidates << File.join(source_dir, basename)
	rpaths.each do |rpath|
		resolved_rpath = rpath.sub(/\A@loader_path/, source_dir).sub(/\A@executable_path/, executable_dir)
		candidates << File.join(File.expand_path(resolved_rpath), basename)
	end
	candidates << File.join(@new_dylib_path, basename)

	candidates.find { |candidate| File.file?(candidate) }
end

Comment on lines +11 to +13
def parse_rpaths(otool_l_output)
otool_l_output.scan(/cmd LC_RPATH\n\s*cmdsize \d+\n\s*path (.+?) \(offset \d+\)/).flatten
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current regular expression for parsing otool -l output assumes strict newline (\n) and spacing formatting. To make the parsing more robust against variations in whitespace, tabs, or line endings (e.g., carriage returns) across different macOS/Xcode versions, it is safer to use \s+ instead of literal newlines and spaces.

def parse_rpaths(otool_l_output)
	otool_l_output.scan(/cmd\s+LC_RPATH\s+cmdsize\s+\d+\s+path\s+(.+?)\s+\(offset\s+\d+\)/).flatten
end

Comment on lines +16 to +18
def rpaths_for_file(file)
parse_rpaths(`otool -l "#{file}" 2> /dev/null`)
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

Using string interpolation directly inside backticks (otool -l "#{file}") can lead to shell injection or command failures if the file path contains spaces, double quotes, or other shell metacharacters. It is highly recommended to escape the file path using Shellwords.escape to ensure the command executes safely and robustly.

def rpaths_for_file(file)
	require "shellwords" unless defined?(Shellwords)
	parse_rpaths(`otool -l #{Shellwords.escape(file)} 2> /dev/null`)
end

On 2026-09-27 homebrew-core moved the `openssl` alias from openssl@3 to
openssl@4 (Homebrew/homebrew-core 9a12b94a). build.sh installs
`openssl@3`, but ruby_ship_build.sh resolved `brew --prefix openssl`,
which now returns /opt/homebrew/opt/openssl@4 -- a directory that does
not exist on the bots. The resulting `-L/opt/homebrew/opt/openssl@4/lib`
makes ld emit "directory not found" warnings, Ruby 3.1's configure
treats any warning under -Werror as a failed probe (e.g. "function name
string predefined identifier... no"), and the build then dies with
`use of undeclared identifier 'RUBY_FUNCTION_NAME_STRING'`.

Seen on led build https://ci.chromium.org/b/8669384638160948033; the
previous prod run (2 days earlier) still resolved the alias to openssl@3.
Pin the prefix to openssl@3 in both places it is used, matching what
build.sh installs and the openssl@3 cert.pem path it already copies.

Related: flutter/flutter#164665
@okorohelijah
okorohelijah marked this pull request as ready for review September 28, 2026 22:34
bundler 4.0.21 (released 2026-09-16) requires Ruby >= 3.2 and RubyGems
>= 3.4.1. `gem install -f bundler` skips that check, so it was force-
installed onto the Ruby 3.1.3 build, and the `bin/bundler --version`
smoke test in tools/build.sh crashed inside the RubyGems resolver:

  rubygems/resolver/conflict.rb:47:in `conflicting_dependencies':
  undefined method `request' for nil:NilClass (NoMethodError)

Seen on led build https://ci.chromium.org/b/8669383997159272673, right
after the dylib relink and gem installs had all succeeded.

Pin to `~> 2.5.0`, which resolves to bundler 2.5.23 (Ruby >= 3.0,
RubyGems >= 3.2.3; Ruby 3.1.3 ships RubyGems 3.3.26). Note that a bare
`~> 2.5` would resolve to 2.7.2, which already requires Ruby >= 3.2, so
the patch-level constraint is deliberate.

Related: flutter/flutter#164665
- Only expand `@loader_path/` references (and LC_RPATH entries) relative
  to the referencing library's original directory. `@executable_path`
  depends on whichever executable loads the library, which is not known
  when packaging a Homebrew dylib, so such references are matched by
  basename only (original directory, LC_RPATH dirs, already-copied
  dylibs) and `@executable_path` LC_RPATH entries are skipped.
- Make the `otool -l` LC_RPATH regex tolerant of arbitrary whitespace.
- Shell-escape the path passed to `otool -l`.
- Use File.basename.
@okorohelijah okorohelijah added the CICD Run CI/CD label Sep 29, 2026
`gem install activesupport -v 7.0.8` pulls in the newest concurrent-ruby
(1.3.8 today). concurrent-ruby 1.3.5 (2025-01-15) removed its implicit
`require "logger"`, which activesupport 7.0 silently relied on, so the
`bin/pod --version` smoke test fails with:

  activesupport-7.0.8/lib/active_support/logger_thread_safe_level.rb:12:
  uninitialized constant ActiveSupport::LoggerThreadSafeLevel::Logger
  (NameError)

Seen on led build https://ci.chromium.org/b/8669375943591764321, after
the relink, openssl@3 and bundler fixes had taken the build all the way
to the last smoke test.

The package is built into a fresh GEM_HOME, so installing concurrent-ruby
1.3.4 first means RubyGems keeps it when resolving activesupport (which
requires `~> 1.0, >= 1.0.2`; cocoapods-core wants `~> 1.1`, i18n and
tzinfo `~> 1.0`, all satisfied by 1.3.4), and `pod` activates 1.3.4 at
runtime instead of 1.3.8.

Related: flutter/flutter#162341
Installing concurrent-ruby 1.3.4 first was not enough on its own: the
RubyGems bundled with Ruby 3.1.3 (3.3.26) still resolves activesupport's
`concurrent-ruby ~> 1.0` to the newest release and installs 1.3.8 next
to 1.3.4, and at runtime `pod` activates the newest installed version, so
`bin/pod --version` kept failing with

  uninitialized constant ActiveSupport::LoggerThreadSafeLevel::Logger

(led build https://ci.chromium.org/b/8669366981838817409, which shows
both `Successfully installed concurrent-ruby-1.3.4` and, during the
activesupport install, `Successfully installed concurrent-ruby-1.3.8`).
Newer RubyGems (3.6) keeps the already-installed 1.3.4, which is why a
local dry run did not show the problem.

After the cocoapods install, uninstall every concurrent-ruby newer than
1.3.4 (`-a` all matching versions, `-x` executables, `-I` skip the
dependency prompt). `gem uninstall` exits 0 with "is not installed" when
nothing matches, so this is a no-op on RubyGems versions that already
keep 1.3.4.

Related: flutter/flutter#162341
@okorohelijah okorohelijah changed the title [ruby] Resolve @rpath/@loader_path dylib references when relinking WIP: [ruby] Resolve @rpath/@loader_path dylib references when relinking Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant