Skip to content

ci: run only the suites a PR's paths can reach, and 4 xdist workers - #1392

Merged
philmerrell merged 3 commits into
developfrom
claude/test-execution-strategy-543374
Sep 30, 2026
Merged

philmerrell merged 3 commits into
developfrom
claude/test-execution-strategy-543374

Conversation

@philmerrell

Copy link
Copy Markdown
Contributor

Summary

The PR gate ran every suite on every pull request. A two-file docs PR paid the same 7–8 minutes as a backend change, and the backend pytest job alone was 7:02 while every other job finished in under 2 minutes. Two changes:

Fix 1 — run only the suites a PR's paths can reach. A changes job in ci.yml reads the PR's file list from the REST API (no third-party action, no fetch-depth games, renames fed under both names) and pipes it through scripts/ci/classify-changes.sh, which maps paths to the run_* inputs tests.yml already had:

Changed paths Suites
docs/, docs-site/, tui/, root markdown, LICENSE none
backend/ backend, load-test unit
backend/scripts/ + infrastructure (jest asserts the backfill scripts exist)
frontend/ frontend, frontend under --coverage
infrastructure/, RELEASE_NOTES.md infrastructure + a new contracts-only backend job (supply_chain + architecture, the tests that read the CDK tree, ~30 s)
.claude/ infrastructure (one jest test reads the release skill)
.github/, scripts/, anything unrecognised, empty list, workflow_dispatch everything

Fail-open by construction: only the explicit first row runs nothing. Filtering is on the jobs, not the trigger's paths:, so a skipped suite reports as skipped rather than stranding a required check as "Expected" if develop ever gains one (it has none today, which is worth knowing on its own).

Fix 2 — -n logical. With psutil installed, -n auto counts physical cores; the 4-vCPU runner reports 2, so every run started 2 xdist workers (created: 2/2 workers in the logs). logical starts 4.

Guard

backend/tests/supply_chain/test_ci_path_filter.py pins the classifier's rules and fail-open cases, asserts every run_* input tests.yml offers is wired in ci.yml (a new suite cannot silently run nowhere on a PR), and asserts the backend step stays on -n logical.

Also

Verification

  • backend/tests/supply_chain + tests/architecture: 213 passed (32.9 s, which is the contracts job's expected runtime). Root tests/supply_chain: 48 passed.
  • This PR touches .github/, so its own CI run should show the changes job classifying to everything; the first docs-only PR after merge is the real proof of the skip path.

🤖 Generated with Claude Code

philmerrell and others added 3 commits September 30, 2026 09:16
`-n auto` counts physical cores when psutil is installed. The 4-vCPU
ubuntu-24.04 runner reports 2, so the job started 2 xdist workers for
~11k tests (`created: 2/2 workers` in every run's log). `-n logical`
starts 4.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The PR gate ran every suite on every pull request, so a docs-only change
paid the full backend suite (7 minutes on the runner) for a result that
could not have changed.

A `changes` job in ci.yml now reads the PR's file list from the REST API
and pipes it through scripts/ci/classify-changes.sh, which maps each
path to the tests.yml suites that read it. Docs, root markdown and the
TUI run nothing; frontend/ runs the two SPA jobs; infrastructure/ runs
jest plus a new contracts-only backend job (supply_chain + architecture,
the tests that read the CDK tree, about 30 seconds) instead of the full
suite. The mapping is fail-open: an unrecognised path, an empty list or
a workflow_dispatch runs everything, and .github/ and scripts/ always
do.

Filtering is on the jobs, not on the trigger's `paths:`, so a skipped
suite reports as skipped rather than leaving a required check
"Expected" if develop ever gains one.

test_ci_path_filter.py pins the rules and their fail-open cases, and
asserts that every run_* input tests.yml offers is wired on the PR gate
so a new suite cannot silently run nowhere on a pull request.

Also replaces the placeholder Testing doc page, which still said pytest
was not run in CI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ow-ups

Points at #1390 (shard the pytest job with pytest-split) and #1391
(scope moto fixtures at module level; setup is 82% of measured suite
time), with the profile numbers behind both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@philmerrell
philmerrell merged commit 45d8239 into develop Sep 30, 2026
9 checks passed
@philmerrell
philmerrell deleted the claude/test-execution-strategy-543374 branch September 30, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant