Skip to content

CI: run the Firefox/Safari/phone tests alongside Chromium, and skip them on main when a draft run already passed the same tree #3524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gitar-bot merged 5 commits into from
Oct 13, 2026
Merged

CI: run the Firefox/Safari/phone tests alongside Chromium, and skip them on main when a draft run already passed the same tree #3524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b9d51ba
CI: run the Firefox/Safari/phone matrix alongside Chromium, split des…
claude Oct 13, 2026
68c8d19
Merge branch 'main' into claude/project-thread-fvnf7d
BigSimmo Oct 13, 2026
ceb533c
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 13, 2026
7af1daa
CI: let main skip the release browser matrix when a full draft run al…
claude Oct 13, 2026
16bb3a0
Merge branch 'main' into claude/project-thread-fvnf7d
BigSimmo Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 34 additions & 15 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ jobs:
tree_proven_coverage: ${{ steps.tree-proof.outputs.coverage }}
tree_proven_ui: ${{ steps.tree-proof.outputs.ui }}
tree_proven_lighthouse: ${{ steps.tree-proof.outputs.lighthouse }}
# True only when a green full-suite dispatch run already passed every release-matrix leg on
# this exact tree (the merged PR's head). Proved separately because PR runs never run it.
tree_proven_matrix: ${{ steps.tree-proof.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -1919,6 +1922,7 @@ jobs:
startsWith(github.ref, 'refs/heads/release/') ||
(
github.ref == 'refs/heads/main' &&
needs.changes.outputs.tree_proven_matrix != 'true' &&
(
needs.changes.outputs.ui_changed == 'true' ||
needs.changes.outputs.perf_changed == 'true' ||
Expand All @@ -1929,9 +1933,15 @@ jobs:
needs.changes.result == 'success' &&
needs.static-pr.result == 'success' &&
(needs.build.result == 'success' || needs.build.result == 'skipped') &&
(needs.ui-playwright-build.result == 'success' || needs.ui-playwright-build.result == 'skipped') &&
(needs.ui-critical.result == 'success' || needs.ui-critical.result == 'skipped')
needs: [changes, static-pr, build, ui-playwright-build, ui-critical]
(needs.ui-playwright-build.result == 'success' || needs.ui-playwright-build.result == 'skipped')
# Starts as soon as the shared Playwright build is ready, ALONGSIDE Production UI rather than
# after it. Waiting for Production UI bought nothing: these legs only read the shared build,
# and the run still fails on a red Production UI (it is required in its own right). Measured
# 2026-10-13: main run 38352161182 had the build ready at minute 4.1, but every leg waited until
# minute 15.2 for the three Chromium shards and the run ended at 34.1; dispatch runs
# 38369056741 and 38370421833 had one red Chromium shard, so the whole Firefox/WebKit matrix
# was skipped and its verdict deferred to another full round after the fix.
needs: [changes, static-pr, build, ui-playwright-build]
Comment thread
BigSimmo marked this conversation as resolved.
Resolved
runs-on: ubuntu-24.04
# Per ENGINE, not for the whole matrix. Before 2026-09-07 one job ran
# chromium-mockups + firefox + webkit sequentially under `workers: 1` /
Expand Down Expand Up @@ -1992,14 +2002,21 @@ jobs:
# merge (main runs took 36-40 min). Tablet now splits four ways (~12 min
# each by its new timing table) and webkit three ways (19.5 -> 13 min by
# its existing table), so every tablet test runs and main finishes sooner.
#
# 2026-10-13: with the matrix no longer queued behind Production UI, desktop WebKit and
# Firefox were the slowest legs (main run 38352161182: webkit 17.2/13.0/14.2 min of tests,
# firefox 13.4/9.0/8.7). Both now split four ways, which brings them under the iPhone legs
# (their slowest single file, ui-smoke, is ~11 min and sets the floor).
include:
- { engine: chromium, project: chromium-mockups, shard: 1, total: 1 }
- { engine: firefox, project: firefox, shard: 1, total: 3 }
- { engine: firefox, project: firefox, shard: 2, total: 3 }
- { engine: firefox, project: firefox, shard: 3, total: 3 }
- { engine: webkit, project: webkit, shard: 1, total: 3 }
- { engine: webkit, project: webkit, shard: 2, total: 3 }
- { engine: webkit, project: webkit, shard: 3, total: 3 }
- { engine: firefox, project: firefox, shard: 1, total: 4 }
- { engine: firefox, project: firefox, shard: 2, total: 4 }
- { engine: firefox, project: firefox, shard: 3, total: 4 }
- { engine: firefox, project: firefox, shard: 4, total: 4 }
- { engine: webkit, project: webkit, shard: 1, total: 4 }
- { engine: webkit, project: webkit, shard: 2, total: 4 }
- { engine: webkit, project: webkit, shard: 3, total: 4 }
- { engine: webkit, project: webkit, shard: 4, total: 4 }
- { engine: webkit, project: mobile-webkit, shard: 1, total: 5 }
- { engine: webkit, project: mobile-webkit, shard: 2, total: 5 }
- { engine: webkit, project: mobile-webkit, shard: 3, total: 5 }
Expand Down Expand Up @@ -2104,19 +2121,21 @@ jobs:
# includes mockups, whose public flag is baked in, so it builds its own.
# If the producer was out of scope, the wrapper also builds normally.
PLAYWRIGHT_REUSE_BUILD: ${{ matrix.engine != 'chromium' && needs.ui-playwright-build.result == 'success' }}
# "true" only when this main push skipped Production UI because the identical tree already
# passed it on the merged PR (Main tree proof) — the same proof the primary path relies on.
UI_PROVEN_BY_PR: ${{ needs.changes.outputs.tree_proven_ui }}
# When UI changed, production Chromium is proven either by Production UI in this same run
# (it runs whenever UI changed and no proof exists, and the run fails if it fails) or, on a
# main push, by the merged PR's identical tree (Main tree proof). Either way this matrix
# need not repeat it.
UI_CHANGED: ${{ needs.changes.outputs.ui_changed }}
run: |
# Both paths below are exhaustive across playwright.config.ts projects
# for their own path, and disjoint between engines. The fail-safe union
# must equal the COMPLETE project set — a project omitted here would
# vanish from CI silently, so
# `tests/ci-browser-matrix-coverage.test.ts` re-derives both unions from
# this file plus playwright.config.ts and fails closed on any drift.
if [ "${{ needs.changes.outputs.ui_changed }}" = "true" ] && { [ "${{ needs.ui-critical.result }}" = "success" ] || [ "$UI_PROVEN_BY_PR" = "true" ]; }; then
# Production Chromium already passed — in this run, or on the merged PR's
# byte-identical tree. Keep mockup Chromium plus the cross-browser
if [ "$UI_CHANGED" = "true" ]; then
# Production Chromium is proven by Production UI in this run, or on the merged
# PR's byte-identical tree. Keep mockup Chromium plus the cross-browser
# backstop without repeating it.
case "$ENGINE" in
chromium) PROJECTS="--project=chromium-mockups" ;;
Expand Down
14 changes: 7 additions & 7 deletions docs/ci-operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ These changes came out of the 6 October 2026 CI-efficiency review and its handov
which runs repeat work. They don't change tests, assertions, thresholds, budgets, browser
projects, shard counts, required check names, security gates or `main` concurrency.

| Change | PR | What it does | What still always runs |
| ------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proven-tree skip on `main` | #3313 | On a push to `main` whose tree equals the tree the merged PR's own `pull_request` run **recorded testing**, skips unit coverage, Production UI and Lighthouse if every leg of that job passed there. | Static checks, safety, Build + deployment boot smoke, Playwright build, visual baselines, the Firefox/WebKit `release-browser-matrix`, failure routing; the weekly schedule and every dispatch run in full. |
| Lighthouse requirement matches its producer | `ci/lighthouse-required-condition` | `PR required` demands a `Lighthouse budget` success on every event where that job is expected to run (schedule, ordinary dispatch and lockfile-only pushes included). A skip is accepted only for the skip label, baseline-refresh dispatch, drafts/out-of-scope changes, or a proven skip. | The producer, budget, baseline and thresholds are unchanged. |
| PR branch-update reuse | `ci/pr-update-proof` | On `synchronize` of a same-repo PR, reuses an earlier green run of the **same PR**. If the newly tested tree is identical, coverage, UI and Lighthouse carry. If the delta lies entirely outside the build/browser/Lighthouse inputs, UI and Lighthouse carry. | Lint, typecheck, unit coverage (unless the tree is identical), safety, secret scanning, SAST, Build. |
| Scheduled npm cache keys | #3318 | Six scheduled workflows hash `package-lock.json` + `.nvmrc`, the key CI's `npm ci` jobs populate, instead of the lockfile-only key a bare `setup-node` can fill empty. | Every `npm ci`. |
| Merge-queue Gitleaks | #3322 | `merge_group` scans the full immutable `base_sha..head_sha` range with the pinned binary and fails closed on a missing, zero, absent or non-ancestor base. | Push/PR/schedule/dispatch scans, `contents: read`, no persisted credentials. |
| Change | PR | What it does | What still always runs |
| ------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proven-tree skip on `main` | #3313 | On a push to `main` whose tree equals the tree the merged PR's own `pull_request` run **recorded testing**, skips unit coverage, Production UI and Lighthouse if every leg of that job passed there. Also skips the Firefox/WebKit `release-browser-matrix` when a green full-suite `workflow_dispatch` run on the same PR head recorded testing that tree. | Static checks, safety, Build + deployment boot smoke, Playwright build, visual baselines, the release matrix without its own dispatch-run proof, failure routing; the weekly schedule and every dispatch run in full. |
| Lighthouse requirement matches its producer | `ci/lighthouse-required-condition` | `PR required` demands a `Lighthouse budget` success on every event where that job is expected to run (schedule, ordinary dispatch and lockfile-only pushes included). A skip is accepted only for the skip label, baseline-refresh dispatch, drafts/out-of-scope changes, or a proven skip. | The producer, budget, baseline and thresholds are unchanged. |
| PR branch-update reuse | `ci/pr-update-proof` | On `synchronize` of a same-repo PR, reuses an earlier green run of the **same PR**. If the newly tested tree is identical, coverage, UI and Lighthouse carry. If the delta lies entirely outside the build/browser/Lighthouse inputs, UI and Lighthouse carry. | Lint, typecheck, unit coverage (unless the tree is identical), safety, secret scanning, SAST, Build. |
| Scheduled npm cache keys | #3318 | Six scheduled workflows hash `package-lock.json` + `.nvmrc`, the key CI's `npm ci` jobs populate, instead of the lockfile-only key a bare `setup-node` can fill empty. | Every `npm ci`. |
| Merge-queue Gitleaks | #3322 | `merge_group` scans the full immutable `base_sha..head_sha` range with the pinned binary and fails closed on a missing, zero, absent or non-ancestor base. | Push/PR/schedule/dispatch scans, `contents: read`, no persisted credentials. |

### How a skip is proven (`scripts/ci-main-tree-proof.mjs`)

Expand Down
Loading
Loading