Skip to content

Add screen_capture_format attribute to ios_xctestrun_runner - #3021

Merged
keith merged 3 commits into
bazelbuild:mainfrom
erneestoc:erneestoc/screen-recording
Aug 21, 2026
Merged

keith merged 3 commits into
bazelbuild:mainfrom
erneestoc:erneestoc/screen-recording

Conversation

@erneestoc

@erneestoc erneestoc commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a new screen_capture_format string attribute to ios_xctestrun_runner that threads Xcode's PreferredScreenCaptureFormat xctestrun key (added in Xcode 15) into the generated file. Accepted values:

  • "" (default): key is not written, so Xcode's platform default is used — no behavior change for existing users.
  • "screenshots": capture screenshots only.
  • "screenRecording": capture a full screen recording of each UI test run, attached to the resulting .xcresult bundle.

The same value can be set ad hoc for a single run without editing the runner target via --test_arg=--xctestrun_screen_capture_format=<value>.

Setting a non-empty value implies:

  1. Use of xcodebuild test-without-building (the direct xctest path does not read the xctestrun file).
  2. create_xcresult_bundle = True (the capture is only observable in the .xcresult).

Both are documented on the attribute and echoed as note: lines when triggered, matching the existing attachment_lifetime/xcodebuild_args conventions.

Interaction with attachment_lifetime

Screen captures are system attachments, so attachment_lifetime must be "keepAlways" or "deleteOnSuccess" for the capture to survive into the .xcresult — the default "keepNever" discards it. This is called out in the attribute doc, and the runner emits a warning: at runtime when a capture format is requested while the lifetime is "keepNever".

Test plan

  • //test:ios_xctestrun_runner_ui_test — 2 new template-emission cases:
    • test_ios_ui_test_default_screen_capture_format_arg — asserts PreferredScreenCaptureFormat is not emitted when the flag is unset.
    • test_ios_ui_test_screen_capture_format_arg — asserts the key + screenRecording value + SystemAttachmentLifetime=keepAlways are all written when both flags are set.
  • //doc:check_ios passes with regenerated doc/rules-ios.md.
  • buildifier --lint=warn passes clean.
  • Manual end-to-end verification against a UI test on iOS 18.6 simulator: .xcresult produced and opens in Xcode with the .mov video attachment visible in the Attachments panel.
  • Verified locally on Xcode 26.2 / iOS 26.2 simulator: both new cases plus test_ios_ui_test_pass and test_ios_ui_test_attachment_lifetime_arg pass.

@erneestoc erneestoc changed the title Add screen_recording attribute to ios_xctestrun_runner Add screen_capture_format attribute to ios_xctestrun_runner Jul 19, 2026
@erneestoc
erneestoc force-pushed the erneestoc/screen-recording branch 3 times, most recently from 72162ec to 8501095 Compare July 20, 2026 03:08
Comment thread apple/testing/default_runner/ios_xctestrun_runner.template.sh Outdated
Comment thread apple/testing/default_runner/ios_xctestrun_runner.template.sh Outdated
Comment thread apple/testing/default_runner/ios_xctestrun_runner.template.sh Outdated
@keith

keith commented Aug 18, 2026

Copy link
Copy Markdown
Member

lgtm, can you rebase

Threads a `PreferredScreenCaptureFormat` value (added in Xcode 15) into the
generated xctestrun file so UI tests can capture a full screen recording
instead of just screenshots. Accepted values are `""` (default; keep Xcode's
platform default), `"screenshots"`, and `"screenRecording"`.

Setting a non-empty value implies use of `xcodebuild test-without-building`
(the direct `xctest` path does not read the xctestrun file) and implies
`create_xcresult_bundle = True` (the recording is only observable in the
`.xcresult`).

Screen recordings are system attachments, so `attachment_lifetime` must be
`"keepAlways"` or `"deleteOnSuccess"` for the recording to survive into the
`.xcresult`; this is documented on the attribute.
Review feedback:
- Rename the attribute (and --xctestrun_* flag) to screen_capture_format:
  it is a passthrough for Xcode's PreferredScreenCaptureFormat key, and
  screen_recording = "screenshots" read as a contradiction.
- Emit a runtime warning when a capture format is requested while
  attachment_lifetime is "keepNever", since the capture would be silently
  discarded before reaching the .xcresult bundle.
- Drop the screenshots-value test: it exercised the identical code path as
  the screenRecording test and the attr values list already guarantees the
  value threads through.
- Restore the pre-existing attachment-lifetime comment so the diff stays
  purely additive.
Review feedback: screen captures are system attachments, so with the
default attachment_lifetime of "keepNever" the capture would be silently
discarded after paying for the xcodebuild path and an xcresult bundle.
Fail fast with a clear error instead of warning, and cover the failure
in the runner test suite.

The simulator serialization commits moved to their own PR (simulator
pool), also per review feedback.
@erneestoc
erneestoc force-pushed the erneestoc/screen-recording branch from 73046fb to bdd8b97 Compare August 19, 2026 18:27
@erneestoc

erneestoc commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main — the conflict was with test_environment (#3045), which landed in the same attribute block; doc/rules-ios.md regenerated with bazel run //doc:update.

CI note, correcting my earlier read of this: 8-dot-0-1-on-darwin-macos-arm64 and 9-dot-x-on-darwin-macos-arm64 failed with Bazel exit status 3 (build fine, tests failed or timed out), while 8-dot-x and last-green-bazel-on-darwin-macos-arm64 passed on the same commit. So this is not an analysis or build problem with the change.

I first attributed this to the simulator-concurrency flakiness described in #3026, but that is at most part of the story and I overstated it. #3026 itself fails 9-dot-x with the same exit status, so it plainly does not make that job green on its own. 9-dot-x is also currently red on several other open PRs (#3008, #3015, #3028) and green on others.

What the red ones have in common is that they invalidate the cached results of the simulator tests, so those tests actually execute instead of returning as cache hits — which is the condition #3026 addresses. But that is a correlation across PR contents, not a diagnosis: I have not been able to read the Buildkite logs to confirm which test actually fails, so I would not want the concurrency explanation taken as established.

Happy to re-run, or to narrow this down if someone with Buildkite access can share the failing output.

@keith
keith enabled auto-merge (squash) August 19, 2026 21:15
@keith
keith merged commit 86396a4 into bazelbuild:main Aug 21, 2026
9 checks passed
@erneestoc
erneestoc deleted the erneestoc/screen-recording branch August 21, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants