Skip to content

ci: select Tests jobs per affected component - #147

Merged
raiseCatError merged 5 commits into
devfrom
infra/ci-affected-components
Nov 3, 2026
Merged

raiseCatError merged 5 commits into
devfrom
infra/ci-affected-components

Conversation

@raiseCatError

Copy link
Copy Markdown
Owner

Summary

The Tests workflow's classifier had two answers: docs/press-kit only (skip everything) or full CI. Every other change ran all four heavy jobs, including the 25-minute iPhone Duo UI-test job on a macOS runner. A release-tooling-only change like #146, a one-line Mac test fix like #143 and an Android-only fix like #68 each used about 36 macOS runner-minutes and took about 26 minutes of wall-clock time.

This PR replaces the two booleans with one output per job, chosen from a dependency map I checked against the real build graph (project.yml targets, the Android Gradle build and the files the macOS tests read at runtime). It doesn't use directory names alone.

Area changed Jobs that run Why (evidence)
Mac/, MacReceiver/, MacTests/ mac (without the iOS build step) No iOS target compiles them (project.yml)
iOS/ mac + iOS build + ios-duo MacTests compiles iOS/DemoSession.swift and reads iOS sources, catalogs and the privacy manifest
iOSUITests/ ios-duo Only the UI-test target compiles it
Shared/ (protocol, pairing, encryption) every app job Every Apple target compiles it; Android also runs as policy
fixtures/ mac + android Both replay the vectors (MacTests and core/build.gradle.kts)
project.yml, .xcode-version every Apple job
AndroidReceiver/ android Nothing outside it reads it
scripts/release, scripts/reference, the 14 listed workflows, rulesets, fastlane config, Gemfile, version files release-tooling tests.yml calls none of them; release-tooling checks them
Docs, issue forms, PR template, FUNDING.yml, README images, press kit nothing heavy (press-kit for press-kit paths) As before
tests.yml, scripts/ci/**, any unlisted path full CI Fails closed

release-tooling also runs for every change that isn't docs or media. It takes about 0.5 minutes on Linux and runs the dependency checks below.

The macOS tests scan the repository for the upstream App Store listing ID, so any changed file that contains it also gets mac, wherever it lives (this used to be checked for docs only).

Scope and compatibility

  • Components: .github/workflows/tests.yml and scripts/ci/ only. No app code, no release workflow, no rulesets.
  • Required checks: mac, release-tooling, android and conventional keep their names. They are never removed by a workflow-level path filter: a job that isn't needed is skipped, and a skipped job reports success. All three Tests jobs still exist on every run.
  • Fail-closed rules:
    • A job is skipped only when the classifier says exactly false for it.
    • Every job runs when the changes job fails.
    • The base branch's classifier decides. A base copy that predates --github, a missing base classifier, a classifier crash or missing HEAD^1 all mean full CI.
  • Event rules: pushes to main and release/**, multi-commit or force pushes to dev, and (new) PRs into main or release/* all run full CI. ci: give each dev push its own Tests run #145's per-SHA dev concurrency is unchanged.
  • Earlier tests.yml copies: the old CLI still works and still writes docs_media_only, workflow_only and press_kit. PR branches that haven't merged dev yet get answers that are safe for their workflow: app changes get full CI.

Verification

Check Result and evidence
Automated tests / lint / build python3 -m unittest discover -s scripts/ci: 75 tests OK. ruby scripts/release/check-release-workflows.rb: fail-closed OK. Release-tooling Ruby suite: 112 runs, 0 failures. YAML parses. bash -n passes on the extracted step script.
Mutation checks Breaking a rule makes the tests fail: fixtures without android, Shared/ without android, project.yml without iOS jobs, MacTests without mac, no upstream-ID scan, PRs into release/* classified, multi-commit dev pushes classified. Breaking a dependency also fails them: iOS target compiling a Mac file, a new Xcode target, MacTests reading docs/*.md, PROTOCOL.md or AndroidReceiver/, Gradle reading ../docs, a new workflow, tests.yml using a local action, a fail-open job condition, a dropped output.
Step simulation The real changes step script ran against scratch clones. Transition PR on today's dev: full CI ("predates per-job selection"). Mac-only PR: mac + release-tooling. Single-commit iOS dev push: Apple jobs. Multi-commit dev push, main push, PR into release/2.1.0, shallow clone and a crashing classifier: full CI.
Physical devices tested None. CI-only change.
Simulator or emulator None locally.
Not tested / known limitations This PR's own run is full CI by design: it changes tests.yml and scripts/ci, and the base classifier predates --github. Targeted selection only takes effect for PRs opened against dev after this merges.

Safety and review checklist

  • This is a focused change with no unrelated refactors.
  • Relevant automated checks ran; failures or skipped tests are documented.
  • I distinguished real-device verification from simulator/emulator or unit tests.
  • Existing pairing, security and consent behavior is preserved, or changes are explained.
  • Public screenshots, logs and test data contain no personal data, private endpoints, identifiers, secrets or pairing codes.
  • Documentation, user-facing terminology and accessibility were updated if relevant.
  • No signing material, credentials, provisioning profiles or local build artifacts are included.
  • This PR does not tag, release, publish, or alter protected release workflows without explicit maintainer coordination.

Notes for reviewers

Measured job durations (medians over 46 to 54 successful jobs in the last 60 Tests runs): ios-duo 25.6 min (UI tests 22.9, simulator boot 1.9), mac 10.2 min (macOS tests 6.3, iOS build 2.3, Mac receiver 0.9), android 5.8 min, release-tooling 0.5 min, press-kit 0.7 min, changes 0.1 min. Both macOS jobs run on xcode-27.

Replay estimate. I replayed the last 80 merged PRs into dev (#34 to #146) through dev's current rules and the new ones, priced at the medians above. "mac without the iOS step" is estimated at 7.7 min. These are estimates, not measured runs.

Current rules New rules
macOS job-minutes (one run per PR) 2,506 1,299 (−48%)
Linux job-minutes 451 271
Sum of critical-path wall-clock 1,801 min 1,031 min

Each PR normally runs Tests at least twice (once per push, plus its merge to dev), so real totals scale up from these figures.

Billing. The repository is public. The runs-timing API reports no billable minutes for these runs, so the gain is runner capacity, macOS queueing and wall-clock time rather than a direct charge.

Intentionally unchanged:

  • The duplicate iOS build in mac (generic device, for the SDK gate) and in ios-duo (simulator). They target different SDKs and can't share artifacts.
  • No DerivedData or SPM cache. Setup steps take about 15 seconds; the cost is compilation and tests.
  • Multi-commit dev pushes still run full CI.
  • Prototypes/, assets/ (other than README images and the press-kit preview), tools/ (other than press-kit), generate.sh and run.sh stay full CI.

@raiseCatError
raiseCatError marked this pull request as ready for review November 3, 2026 21:50
@raiseCatError
raiseCatError merged commit 844ee6a into dev Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant