Skip to content

Reduce PR CI to 26 jobs and avoid duplicate Python tests - #10007

Merged
hua7450 merged 5 commits into
PolicyEngine:mainfrom
hua7450:ci-yaml-only-test-batches
Oct 8, 2026
Merged

hua7450 merged 5 commits into
PolicyEngine:mainfrom
hua7450:ci-yaml-only-test-batches

Conversation

@hua7450

@hua7450 hua7450 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Closes #9904.

Reduce the Pull request workflow from 35 to 26 runner jobs, organize Baseline, Contrib and Python tests into collapsible groups with separate runners, and remove duplicate Python execution from YAML batches.

Work Before After
Registry lock/rehearsal, release guard tests, lint, changelog validation 4 1
Python compatibility (3.11–3.14), rc1/final wheels, bundle metadata contract 7 1
Quick Feedback 1 1
Baseline 11 11
Household API Partners 1 1
Contrib 9 9
Rest and Microsimulation, grouped as Python tests 2 2
Total 35 26

Execution and isolation

The validation job retains independent step results. Later checks run after earlier failures with !cancelled(); failures still fail the job, and cancellation stops further checks. Downstream jobs wait for all four validation areas, so lint/changelog failures also block them and the validation gate can take longer than the old lock-only gate.

Package and compatibility runs sequentially on one Ubuntu 24.04 runner: rc1 wheel, final wheel, bundle metadata, then Python 3.11–3.14. Each area has a separate checkout and environment. Python versions retain independent lock resolution, smoke imports and dataset copy/extension tests. Wheels still build the reviewed PR head, with the original Python 3.14.7/uv 0.12.13 pins, source-integrity checks, artifact names, receipts and retention. Metadata retains its pinned bundle dependency and original test commands. An earlier failure does not suppress later areas; actual failures remain failures.

Rest and Microsimulation move into a reusable workflow called Python tests. This groups them in the Actions view while preserving two parallel jobs on two runners, with unchanged commands, process boundaries, environments, 90/60-minute timeouts, and separate timing/memory artifacts. Baseline and Contrib each move their existing matrix into a matching reusable workflow, creating collapsible Baseline tests (11 jobs) and Contrib tests (9 jobs) groups. Their commands, matrix entries, failure behavior and timeouts are unchanged. All three callers allocate no runners, and the reusable workflows have no independent PR/push triggers. Every runner job remains a separate PR check.

Draft PRs still allocate zero runners. Draft guards, ready/draft triggers and cancellation are unchanged; there is no cleanup job. GitHub may display skipped checks. Quick Feedback, Partners and the push/release workflow remain unchanged; Baseline and Contrib retain their existing job definitions inside the new groups. The main branch required-check list and effective branch rules were empty when inspected; no protection settings were changed.

Duplicate test execution

YAML batch subprocesses exclude .py collection, keeping those tests in their dedicated Python steps. The two Python-only baseline subdirectories no longer create empty YAML batches. Every batch containing YAML keeps its original paths, grouping and memory budget. No existing policy assertions or example counts are removed.

In completed CI run 37687392817, nine additional-tax-bracket cases consumed 288.99 seconds inside Contrib other-shard-1's YAML step and ten AFA cases consumed 270.98 seconds inside Congress's YAML step, then both files ran again in the Contrib Python step. The collection fix removes approximately 9m20s of duplicated runner work observed there. The routing audit also identified 15 baseline Python files exposed to duplicate collection, all retained in Rest.

Savings and tradeoffs

This removes nine runner allocations per full PR run. Packaging/compatibility now takes the sum of its sequential checks and rerunning its failed job repeats more work. Historical timings put those areas together at roughly eight minutes; that is a planning estimate, not a measured combined-job result. Overall wall time, queue time, disk use and runner-minutes need confirmation in normal Linux CI. Job count alone does not establish equivalent runner-minute savings.

Validation

  • Static audit: 26 expanded runner jobs; the three reusable workflow wrappers allocate none. Baseline, Contrib, Rest and Microsimulation definitions are identical except that their validation dependency is now on their callers. Bundle metadata commands are identical.
  • 11 lightweight workflow checks passed, covering the runner budget, draft guard, all three reusable workflow groups, separate Python runners, supported versions, clean environments, failure/cancellation handling, and local make test coverage of reusable-workflow suites.
  • Three affected timing/report-routing checks passed; existing policy test suites were not rerun for their move.
  • 43 release-tool tests passed with RELEASE_LOCK_REAL_UV=1, including real uv fixtures and wheel/publication contracts (1.33s locally).
  • Actionlint 1.7.12 passed for the PR and all three reusable workflows, plus wrapper validation of all three composite actions' steps.
  • Prior collection validation: 43 focused batch-runner checks passed; both collection regressions failed before the fix. All seven Contrib Python files and all 17 affected Python files retain their dedicated routes.
  • make format, git diff --check, and a top-level changelog fragment.
  • Full CI timing and Actions grouping display: verify using the normal ready-for-review run.

The branch includes merged #10006, retaining its Rest setup savings.

axiom: n/a: CI orchestration and test collection only; no policy changes.

@hua7450 hua7450 changed the title Keep Python suites out of YAML test batches Reduce PR CI to 28 jobs and avoid duplicate Python tests Oct 8, 2026
@hua7450 hua7450 changed the title Reduce PR CI to 28 jobs and avoid duplicate Python tests Reduce PR CI to 26 jobs and avoid duplicate Python tests Oct 8, 2026
@hua7450
hua7450 marked this pull request as ready for review October 8, 2026 01:35
@hua7450
hua7450 merged commit 023cb82 into PolicyEngine:main Oct 8, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore consolidating short PR CI jobs to reduce runner allocations

1 participant