Repository navigation
Run the Rest job's remaining Python tests in six pytest processes - #9787
Merged
Merged
Conversation
As one pytest process, the remaining Python tests peaked at 15.7 GB RSS on the 16 GB runner (run 37198184683), with 1.48M major page faults, and since #9782 the step has been cancelled partway through on several runs, with no log retained. test-other-python-rest now runs six groups in sequence, one pytest process each. cached-arrays, heavy-a and heavy-b hold the seven slowest modules; then come core/, policy/, and remaining, which takes every other file, including new ones. The target runs every group even after one fails, and fails if any did. With REST_REPORT_DIR set, as both workflows set it, each group writes its own JUnit XML and GNU time report, and the Rest job uploads them all. The SPM step, the --ignores and the test set are unchanged. code_health/test_rest_python_groups.py takes the old single command and each group's command from `make -n`, lists the files pytest collects for each without importing them, and checks that the groups collect every file of the old command exactly once. It also checks that every group's reports are uploaded by both workflows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the split found that two core modules added after run 37198184683 are heavy too. Scaled to that run's speed, the timings from runs 37198179476 and 37196338557 put test_marginal_tax_rate_coverage at about 124 s and test_dependent_net_investment_income_invariants at about 64 s. The core group would have reached about 330 s. Move the first into heavy-b and the second into heavy-a. That gives 403 / 350 / 345 / 142 / 185 / 239 s across the six groups. Correction to the previous commit's message: the failed runs did keep logs. Four runs on 2026-10-04 (37201566995, 37201944922, 37202679009, 37205594473) end with "The runner has received a shutdown signal" and make's "test-other-python-rest] Killed". That came after 1,530-1,560 passed tests, inside test_formulas_do_not_write_into_cached_arrays or at the start of test_md_poverty_line_credit_invariants. The Makefile comment now cites them. The guard gains two checks: - It runs the real target with a stub pytest, failing one group and SIGKILLing another. Every group must still run, with the dry run's arguments, and the target must fail, naming both groups. - Test file names must stay unique outside packages. The old single process failed on a clash, and separate groups would not. The changelog now states the memory bound as the aim, since the first CI run will be the first to measure each group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second-round review of the split: - The stub-pytest test failed only heavy-a and core. A `-` prefix on any other group's recipe would have hidden that group's failures. An all-fail case now exercises every recipe. A mutation check on cached-arrays, heavy-b, policy and remaining failed it, as it should. - The duplicate-name check now compares the module name that pytest's default import mode gives each file: its dotted path inside a package, or its stem outside one. - Correction to the previous commit's message and the Makefile comment: the four runner losses came 1,424-1,454 tests into the test-other-python-rest step. The 1,530-1,560 figure counted the SPM step's 106 tests as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
First CI run on this head (run 37219969714). Full Suite - Rest passed in 58 minutes. Per-process peak memory, from each group's
Before the split, the remaining Python tests ran as one process that peaked at 15.67 GB, with 1.48M major page faults (run 37198184683). The largest group now peaks at 10.93 GB, with no paging. The job's wall time is 58 minutes, close to the 60-minute limit, so #9785 (limit raised to 90) is still worth landing after this. |
Contributor
Author
|
Merged by the US + core hub on the merge gates:
|
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs the
Full Suite - Restjob's remaining Python tests as six pytest processes in sequence, one per group. A process's exit releases its heap before the next group starts.Why
rest-test-timings), the single process peaked at 15.7 GB RSS on the 16 GB runner, with 1.48M major page faults.make … Killed. Each died 1,424–1,454 tests into the step, insidetest_formulas_do_not_write_into_cached_arraysor at the start oftest_md_poverty_line_credit_invariants.Change
Makefile.test-other-python-restruns six groups:cached-arraystest_formulas_do_not_write_into_cached_arraysheavy-aheavy-bcorecore/policypolicy/remainingTimes are at the measured run's speed; the two newer heavy modules were timed in later runs. Every group runs even if an earlier one fails, and the target fails, naming the groups, if any did.
Reports. With
REST_REPORT_DIRset (bothpr.yamlandpush.yamlset it), each group writesrest-python-<group>.xmlandrest-python-<group>-resources.txt(GNU time), and the Rest job uploads them all. The first CI run on this PR reports per-group peak memory.Guard test (new).
policyengine_us/tests/code_health/test_rest_python_groups.pychecks that:The guard was mutation-checked: dropping the policy group,
|| true, stopping at the first failure, mismatched report paths and more were all caught.Unchanged: the test set, the
--ignores (contrib, microsimulation, SPM files), the SPM step andtimeout-minutes.Verification
uv run --no-sync pytest policyengine_us/tests/code_health/test_rest_python_groups.py: 10 passed.make -n test-other-python-restprints the six group commands.make formatis clean.Before merge: read the per-group
resources.txtfrom this PR's CI run, to confirm each group's peak memory is well under 16 GB.Related:
pr.yaml.make test) was checked against this change for clashes.axiom: n/a: CI configuration only.
🤖 Generated with Claude Code