Skip to content

Retry failing iOS E2E flows once and stop rerunning real failures - #59635

Open
cipolleschi wants to merge 1 commit into
mainfrom
cipolleschi/e2e-ios-1-retry-policy
Open

cipolleschi wants to merge 1 commit into
mainfrom
cipolleschi/e2e-ios-1-retry-policy

Conversation

@cipolleschi

@cipolleschi cipolleschi commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Stack (iOS RNTester E2E cost, review bottom-up):

  1. Retry failing iOS E2E flows once and stop rerunning real failures #59635 ← this PR
  2. Cancel superseded Test All runs on pull requests #59636
  3. Run iOS E2E flows in a single Maestro invocation #59637
  4. Run the FlatList maintainVisibleContentPosition E2E flows in Release only on iOS #59638
  5. Deep-link into the FlatList maintainVisibleContentPosition example in E2E flows #59639
  6. Overlap simulator boot with the rest of the iOS E2E setup #59640
  7. Fail iOS RNTester E2E straight away when the app artifact is missing #59641
  8. Run iOS RNTester E2E tests on Apple Silicon runners #59642

Summary:

Part 1 of a stack that cuts the cost of the iOS RNTester E2E jobs (2 × macos-26-large, ~80 min each per run). Over Oct 27–29, 48% of Test All runs needed a workflow-level retry of these jobs and 44% of their macOS minutes went to those retries. The biggest offender was sectionlist-viewability: it failed ~88% of attempts for over a day, and the retries let some runs pass anyway.

Each flow had 5 attempts inside the job, and the whole job (both flavors, every flow, all setup) was retried up to 3 times on top. That's up to 15 runs of a broken flow. The runner also stopped at the first flow that used up its attempts, so later failures stayed hidden.

This PR:

  • Gives each flow 2 attempts, rebooting the simulator before the retry. The infrastructure failures in the logs (SpringBoard crash, "Unable to launch app", driver timeouts) left the simulator unusable for every later attempt on it.
  • Keeps running the remaining flows after a failure, reports every failed flow at the end and writes a step summary.
  • Persists per-flow results between CI attempts (same format as the Android runner), so retries skip flows that already passed. A flavor that already passed doesn't run again.
  • Only retries the workflow when a job stopped before running every flow (retryable output). A flow that failed both attempts fails the run immediately.
  • Uploads the recording, JUnit report and Maestro debug output for failed attempts only (previously only the last flow's recording per attempt number survived).

The template app E2E shares the action and script. It gets the new in-job behavior (2 attempts, continue past failures) but keeps its own workflow retries unchanged.

Changelog:

[INTERNAL] [CHANGED] - Retry failing iOS E2E flows once and stop rerunning real failures

Test Plan:

  • yarn jest .github/workflow-scripts/__tests__/maestro-ios-test.js: 9 tests pass, covering continuing past failures, one retry with a simulator reboot, and skipping flows that passed in an earlier CI attempt.
  • Ran the script locally with Maestro 2.6.1 against RNTesterApp-NewArch-Release from run 40888922130, on an iPhone 17 Pro / iOS 26.5 simulator, with appearance, button, image-wide-gamut and a flow that always fails. The three passing flows ran once. The failing flow ran twice with a reboot in between, then the script exited 1. The state file recorded passed/failed and attempt counts, and /tmp/MaestroLogs only contained the failing flow's recording, report and debug output.
  • actionlint reports no new issues for the changed workflows.
  • CI on this PR exercises the workflow changes.

Each RNTester iOS flow had 5 attempts inside the job, and the whole job
(both flavors, every flow, all setup) was retried up to 3 times on top, so
a flow could run 15 times. The runner also stopped at the first flow that
exhausted its attempts, hiding later failures.

- Give each flow 2 attempts, rebooting the simulator before the retry.
- Keep running the remaining flows after a failure and report every
  failed flow at the end, with a step summary.
- Persist per-flow results between CI attempts (same format as the
  Android runner) so retries skip flows that already passed.
- Only retry the workflow when a job stopped before running every flow.
  A flow that failed both attempts fails the run immediately.
- Upload the recording, JUnit report and Maestro debug output for failed
  attempts only.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. p: Facebook Partner: Facebook Partner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant