Skip to content

fix(desktop): take desktop tests off the master push lane - #96633

Merged
trunk-io[bot] merged 4 commits into
masterfrom
posthog/desktop-cut-macos-runner-demand
Sep 8, 2026
Merged

fix(desktop): take desktop tests off the master push lane#96633
trunk-io[bot] merged 4 commits into
masterfrom
posthog/desktop-cut-macos-runner-demand

Conversation

@gantoine

@gantoine gantoine commented Sep 8, 2026

Copy link
Copy Markdown
Member

Problem

  • A desktop pull request or merge queue batch waits up to 76 minutes for a macOS test that runs in 6 minutes. Run 34207978083 queued 76 minutes, ran 5.9, and passed.
  • depot-macos-26 is a fixed pool shared across Depot customers, with no autoscaling, because Apple requires a 24-hour minimum lease. We can change how often we join that queue, not our position in it.
  • The waits are not self-inflicted. For 71 of those 76 minutes, no PostHog macOS job was running at all. The most PostHog macOS jobs seen running at once, over three weeks, is two.
  • Master pushes make most of the requests. Every products/desktop/** push asks for two macOS runners: the integration test and the macOS cache warmer.

Jobs on depot-macos-26 over 24 hours, from the GitHub jobs API:

Lane Jobs Run minutes Queued minutes
master push, desktop-test integration test 24 135 337
master push, desktop-warm-caches 23 87 287
pull request 8 63 0
release tag 4 45 0
merge queue batch 3 20 1
master push, desktop-update-e2e 1 9 0

Changes

  • desktop-test.yml stops running on master pushes. A daily 05:03 UTC cron carries master coverage instead. Nothing dispatches on a desktop push, so the Linux jobs stop repeating the batch as well.
  • The merge queue batch tests macOS again. This reverts fix(desktop): skip the macOS integration test on merge queue batches #96546, so the batch keeps covering the merged tree.
  • desktop-warm-caches now fires only when products/desktop/pnpm-lock.yaml changes. Every cache key in that workflow hashes that one file, and its install steps already no-op when all three caches hit, so the other runs took a macOS runner to learn they had nothing to do.

Across the last 7 days the master lane drops from 100 macOS jobs, from 50 pushes, to 23: 7 scheduled plus 16 lockfile changes.

The diff removes more lines than it adds. Skipping the workflow beats guarding the macOS job by event, which would leave the run dispatching and the Linux jobs repeating work the batch already did.

Note

The e2e job needs integration-test, so the live-model gateway suite follows this schedule on master. It still runs on every pull request and every batch. Its pnpm cache save keys off github.ref, which a cron run still satisfies.

Before:

flowchart LR
    A["desktop PR"] --> M1["macOS integration test"]
    B["merge queue batch"] --> S1["skipped by #96546"]
    C["master push"] --> M2["macOS integration test"]
    C --> M3["macOS cache warmer"]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class M1 phBlue;
    class M2,M3 phRed;
    class S1 phGray;
Loading

After:

flowchart LR
    A["desktop PR"] --> M1["macOS integration test"]
    B["merge queue batch"] --> M2["macOS integration test"]
    C["master push"] --> S1["no run"]
    D["daily 05:03 UTC"] --> M3["macOS integration test"]
    E["pnpm-lock.yaml push"] --> M4["macOS cache warmer"]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class M1,M2 phBlue;
    class M3,M4 phYellow;
    class S1 phGray;
Loading

How did you test this code?

  • No tests were added. The change is a trigger and a job condition, and this repo has no harness that evaluates a GitHub Actions if: expression.
  • actionlint, bin/hogli lint:workflows and bin/hogli ci:preflight all pass on this branch.
  • The queue and run times come from the jobs API over recent runs of desktop-test.yml, desktop-ci.yml, desktop-warm-caches.yml, desktop-update-e2e.yml and desktop-release.yml. Push frequency comes from git log over origin/master.
  • The workflow was read for anything else the push lane carried. Only the e2e job's pnpm cache save is ref-scoped, and a cron run keeps github.ref at refs/heads/master.
  • Not verified: that the cron fires, and that a desktop master push now dispatches nothing. Both need a run on this trigger, which exists only once this lands.
  • No manual testing. A workflow trigger cannot be exercised locally.

Automatic notifications

  • Publish to changelog?

Docs update

None. The change affects CI scheduling only.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Raised from a Slack thread about slow desktop tests. The thread's working diagnosis was contention on the macOS pool, so the first fix cut merge queue demand (#96546). The job timings then showed the long waits happen while PostHog's own macOS jobs sit idle, which moved the target from the batch lane to the master-push lane, and made reverting #96546 part of the fix rather than a reversal of it.

Skills invoked: /authoring-ci-workflows and /writing-pr-descriptions.

Two rounds of review from the driver shaped the result. The first draft guarded the macOS job with github.event_name != 'push' and kept the trigger, which still dispatched a run per desktop push; skipping the workflow replaced it. The cron started at six-hourly, until measured push frequency made that a 44% cut.

On cadence: the hourly master lane in /authoring-ci-workflows is the documented pattern for this substitution, and hourly cannot work here, because 24 runs a day is what this PR removes. Daily master crons are already common in this repo, including for suites: ci-hobby.yml at 06:00, ci-e2e-playwright-audit.yml at 07:00, ci-storybook-update-test-timing.yml at 04:00, and desktop-update-e2e.yml, which runs a depot-macos-26 job daily at 07:00. ci-geoip-canary.yml runs six-hourly. So the cadence is in-house, but this is the first daily lane that replaces a suite's master-push coverage.

desktop-test.yml is in neither GATING_WORKFLOWS nor SCHEDULED_GATING_WORKFLOWS in ci-alerts-devex.yml, so no desktop lane pages today. Registering the new cron there is a follow-up: the alerter's scheduled thresholds assume an hourly cadence, and a daily lane needs its own arms.

No duplicate: #95852 and #94509 are open drafts on the same pool. Both cut the time a job spends on a macOS runner; neither cuts how many runners the repo asks for.

The diff carries no customer data or other non-public material.

@gantoine gantoine added skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs feature/desktop Feature Tag: Desktop labels Sep 8, 2026
@gantoine gantoine self-assigned this Sep 8, 2026
@trunk-io

trunk-io Bot commented Sep 8, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

@gantoine
gantoine marked this pull request as ready for review September 8, 2026 13:08
Copilot AI lite review requested due to automatic review settings September 8, 2026 13:08
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team September 8, 2026 13:09
gantoine and others added 2 commits September 8, 2026 09:11
The macOS integration job and the macOS cache warmer both fired on every
products/desktop/** master push, which was 48 of the last 63 jobs on the
depot-macos-26 pool. The merge queue batch already tests the tree that
lands, and every warm-cache key hashes pnpm-lock.yaml alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gantoine
gantoine force-pushed the posthog/desktop-cut-macos-runner-demand branch from da6b745 to d114122 Compare September 8, 2026 13:12
@gantoine gantoine changed the title fix(desktop): cut macOS runner requests from desktop master pushes fix(desktop): take desktop tests off the master push lane Sep 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new scheduled lane’s intent is partially undermined by missing DevEx alerter registration (and there’s also a cron/comment mismatch that should be corrected).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adjusts Desktop CI workflow triggers to reduce contention on the fixed depot-macos-26 runner pool by eliminating macOS runner usage on master pushes, while preserving macOS coverage via the merge queue lane and a scheduled post-merge run.

Changes:

  • Limit desktop-warm-caches to run only when products/desktop/pnpm-lock.yaml (or the workflow) changes.
  • Add a daily scheduled run for desktop-test and isolate its concurrency group from push runs.
  • Skip the macOS integration-test job on master push events (so macOS coverage comes from PRs/merge queue + schedule instead).
File summaries
File Description
.github/workflows/desktop-warm-caches.yml Narrows push trigger paths so cache warming only consumes macOS capacity when the lockfile (cache key input) changes.
.github/workflows/desktop-test.yml Adds a scheduled post-merge lane and updates concurrency; prevents macOS integration tests from running on master pushes.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/desktop-test.yml
Comment thread .github/workflows/desktop-test.yml Outdated
@greptile-apps

greptile-apps Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Fix all with Greploop Fix All in Claude Code

Prompt To Fix All With AI
### Issue 1
.github/workflows/desktop-test.yml:30
**Concurrency key can collide**

The literal `scheduled` key can match a pull request whose branch is also named `scheduled`. Because pull request runs cancel in-progress jobs in the same group, that branch could cancel the daily macOS run and remove the post-merge coverage this change relies on. Use a namespaced key that cannot overlap with a branch name.

```suggestion
    group: desktop-test-${{ github.event_name == 'schedule' && 'event-scheduled' || github.head_ref || github.ref }}
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: da6b745 | Re-trigger Greptile

Comment thread .github/workflows/desktop-test.yml Outdated
gantoine and others added 2 commits September 8, 2026 10:02
The schedule and paths comments explained themselves against the triggers
they replaced, which reads as change history once the diff is merged. State
why each trigger is shaped the way it is instead, and drop the "emptiest hour
measured" claim, which nothing keeps current.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A draft desktop pull request asked for a depot-macos-26 runner on every
push. That pool is fixed and shared between Depot customers, and Apple's
24-hour minimum lease stops it from autoscaling, so the wait for a runner
can be many times the length of the job. A draft now runs quality,
typecheck and unit tests only.

Trunk opens each merge queue batch as a draft pull request, so both
conditions exempt trunk-merge/** heads by name. A bare draft check would
skip the run that gates master.

desktop-ci.yml gains ready_for_review, so marking a pull request ready is
what buys back the build and E2E coverage. Without it the suite would wait
for an unrelated push, and a batch could be the first place either job ran.

The live-model e2e job needs integration-test, so its skip cascades to
drafts as well. Both collation jobs already treat a skipped dependency as
success, so a draft still reports green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@trunk-io
trunk-io Bot merged commit 2cfecf6 into master Sep 8, 2026
199 checks passed
@trunk-io
trunk-io Bot deleted the posthog/desktop-cut-macos-runner-demand branch September 8, 2026 17:45
@deployment-status-posthog

deployment-status-posthog Bot commented Sep 8, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-09-08 18:22 UTC Run
prod-us ✅ Deployed 2026-09-08 18:39 UTC Run
prod-eu ✅ Deployed 2026-09-08 18:41 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/desktop Feature Tag: Desktop skip-agent-review Save $$$, skip auto agent reviews (Greptile) — use for trivial or chore PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants