Repository navigation
Run iOS RNTester E2E tests on Apple Silicon runners - #59642
Open
cipolleschi wants to merge 8 commits into
Open
cipolleschi wants to merge 8 commits into
cipolleschi wants to merge 8 commits into
Conversation
Each RNTester iOS flow had 5 attempts inside the job, and the whole job (both flavors, every flow, all setup) was retried up to 3 times on top, so a flow could run 15 times. The runner also stopped at the first flow that exhausted its attempts, hiding later failures. - Give each flow 2 attempts, rebooting the simulator before the retry. - Keep running the remaining flows after a failure and report every failed flow at the end, with a step summary. - Persist per-flow results between CI attempts (same format as the Android runner) so retries skip flows that already passed. - Only retry the workflow when a job stopped before running every flow. A flow that failed both attempts fails the run immediately. - Upload the recording, JUnit report and Maestro debug output for failed attempts only.
Test All had no concurrency group, so every push to a pull request left the previous run going to completion. Over Oct 27-29 about 40% of the macOS minutes spent on iOS E2E went to runs whose pull request had already received a newer push. Cancel in-progress runs of the same pull request. Push and workflow_dispatch runs get a unique group and are never cancelled.
The runner started a new `maestro test` process for every flow. Each one paid for JVM startup and an XCTest driver reinstall (~24s) plus stopping the screen recording (~3s): about 21 minutes of every ~75 minute RNTester job. - Run every pending flow in one `maestro test` invocation and read the per-flow results from its JUnit report. Flows missing from the report (Maestro crashed or timed out) count as failed. - Retry each failed flow on its own, on a rebooted simulator, with a screen recording. Passing runs don't produce recordings any more; failures keep Maestro's screenshots and debug output. - Add an `exclude-tags` input, and skip the `android-only` flows in RNTester. They only contain an Android-gated `runFlow`, but still paid the per-flow startup cost on iOS.
…only on iOS The 24 FlatList maintainVisibleContentPosition flows are half of the RNTester Maestro suite (~37 minutes per iOS job). Android already runs them against the release build only. On iOS about 88% of their failed attempts over Oct 27-29 were in Debug (133 vs 17). Rename their `android-release-only` tag to `release-only` and exclude it from the Debug job on both platforms.
This was referenced Oct 29, 2026
meta-cla
Bot
added
the
CLA Signed
The iOS E2E action did all of its setup one step after another before touching the simulator. `xcrun simctl install` then waited ~2 minutes for the simulator to finish booting. - Start booting the simulator as the first step (new `boot` subcommand of maestro-ios.js), so it boots while Maestro, Java, the node modules and Metro are being set up. - Drop the `idb-companion` install. Maestro removed its IDB dependency on iOS in 1.x (see "Removed IDB on iOS" in its changelog) and 2.6.1 doesn't reference it. - Drop the second Node.js setup: both callers (RNTester and the template app) already set up Node.js before using the action.
test_e2e_ios_rntester only checks that test_ios_rntester succeeded, but test_ios_rntester uses continue-on-error, so it reports success even when the leg that uploads RNTesterApp-NewArch-<flavor> failed. In run 40709683649 the Release app was never uploaded and the E2E job still started (and retried) three times. Download the app as the first step after checkout, and treat a missing app as a non-retryable failure with an error that points to the build job, instead of an incomplete run that triggers the workflow retries.
The E2E jobs ran on macos-26-large, the Intel x86_64 runner: the iOS
simulator ran under x86_64 and rendered in software ("GL pipe is running
in software mode" in every log), and the -large runners are billed at a
premium.
Run them on the standard arm64 macos-26 runner instead, where the
simulator runs natively with GPU rendering and which isn't billed for
public repositories. RNTesterApp-NewArch-{Debug,Release} are already
universal (x86_64 + arm64 for the app and all its frameworks), so the
build job doesn't need to change.
If the 3-core standard runner turns out to be too slow, macos-26-xlarge
(arm64, 5 cores) is the fallback.
The arm64 runner still logs "GL pipe is running in software mode", so don't claim GPU rendering.
cipolleschi
force-pushed
the
cipolleschi/e2e-ios-8-apple-silicon
branch
from
October 30, 2026 10:57
61952bf to
7d20429
Compare
cipolleschi
force-pushed
the
cipolleschi/e2e-ios-7-artifact-gate
branch
from
October 30, 2026 10:57
0136439 to
23b9b77
Compare
cipolleschi
changed the base branch from
cipolleschi/e2e-ios-7-artifact-gate
to
main
October 30, 2026 11:46
|
@cipolleschi has imported this pull request. If you are a Meta employee, you can view this in D128581888. |
cortinico
approved these changes
Oct 30, 2026
cortinico
left a comment
cortinico
left a comment
Contributor
There was a problem hiding this comment.
Review automatically exported from Phabricator review in Meta.
andrewdacenko
approved these changes
Oct 30, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (iOS RNTester E2E cost, review bottom-up):
Summary:
Part 7 of the iOS E2E cost stack.
The RNTester E2E jobs ran on
macos-26-large, the Intel x86_64 runner, and the-largerunners are billed at a premium.Run them on the standard arm64
macos-26runner instead. The simulator and app run natively on arm64, and standard GitHub-hosted runners aren't billed for public repositories.RNTesterApp-NewArch-{Debug,Release}are already universal (liposhows x86_64 + arm64 for the app binary,hermesvm,ReactandReactNativeDependencies), so the build job doesn't change.Measured on CI (table in the Test Plan), the 3-core standard runner is faster than
macos-26-largedespite having fewer cores: the Debug job took 22.0 min instead of 44.6, and the Release job 43.2 min instead of 55.0. The two jobs run in parallel, so the E2E stage now takes about as long as the Release job (~43 min instead of ~55). Billed minutes go from ~100-largeminutes per run to none. The only step that got slower is the Maestro download and install (~2 min more). If the standard runner ever turns out to be too slow,macos-26-xlarge(arm64, 5 cores) is the fallback. The template app E2E is unchanged.Changelog:
[INTERNAL] [CHANGED] - Run iOS RNTester E2E tests on Apple Silicon runners
Test Plan:
Checked the architectures of the CI artifacts from run 40888922130 with
lipo -info.Ran the RNTester Release artifact locally on an arm64 iPhone 17 Pro simulator with Maestro 2.6.1 (see parts 1 and 3).
CI, comparing the previous run of this PR (
macos-26, arm64) with Fail iOS RNTester E2E straight away when the app artifact is missing #59641 (the same stack, still onmacos-26-large). Both runs still included the since-dropped deep-link change, so the comparison is fair. Single samples, so expect noise.macos-26-large(Fail iOS RNTester E2E straight away when the app artifact is missing #59641)macos-26(Run iOS RNTester E2E tests on Apple Silicon runners #59642)The flows run faster on arm64 even with 3 cores. The only consistently slower step is downloading and installing Maestro (~2 min more); yarn install timing varies between runs on both runners.
"GL pipe is running in software mode" still appears once per job on arm64 (vs 15–18 times per job on Intel), so I'm not claiming GPU rendering.