Repository navigation
Rebalance Rest Python groups within the existing runner budget - #10102
Merged
Merged
Conversation
Contributor
Author
|
US + core hub merge audit, us#10102 at
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rest's
coreprocess rose from 7.1 GB RSS in 38056548883 (base 75cdd80) to 13.3 GB in 38141803955 (#10058 head d692879 tested with main fdd7062). #10058 moved its capital-gains MTR file to heavy-a (8.1 GB, 9:55). Four main additions followed: SSI resource deeming (85.111s), WV senior/disability deduction (31.738s), Schedule A pre-tax premiums (3.813s), and ID health premiums (1.407s). Policy and remaining already measured 13.9 and 14.4 GB on the 16 GB runner.Add three sequential pytest groups on the existing Rest runner, keeping each Python file in exactly one group. The workflow already runs the Makefile group list serially and uploads every group's reports through wildcards. No jobs, matrix entries, or concurrency are added. The routing guard now generates pass/failure/SIGKILL scenarios for the actual group count.
SSI is the only core extraction; the smaller additions stay there. Keep cached-arrays/heavy-a/heavy-b unchanged. Move the retained-model files and three long-running files to shorten remaining; separating NIIT/CTC/Medicaid from NY/AL pins avoids approaching 10 GB before transient allocations.
These are planning estimates requiring Linux CI confirmation. JUnit measures each file's elapsed time; GNU time measures whole-process maximum RSS. Neither measures individual file memory contributions, so the 6.1 GB core rise cannot be assigned entirely to SSI. Source inspection identifies retained copies; the Makefile's roughly 1.45 GB per full parameter-tree copy gives remaining 14.4 − 3×1.45 ≈ 10.1 GB, around the requested 10 GB target, and core-heavy roughly 7.25 GB retained before transient allocations. GC, allocator reuse, and newer main/Core changes make actual peaks uncertain. RSS values follow the supplied convention: reported kbytes / 1,000,000. Wall estimates subtract moved case time; each new process adds roughly 1–2 minutes startup/collection. The longest group falls from 22:18 to an estimated 14:42. #10058 remains open, so its separate heavy-a move is anticipated and not duplicated here.
Follow-up if one file still dominates: SSI builds seven Simulations and duplicates a reformed policy. Reuse one read-only policy where isolation permits and collect released simulation graphs without dropping vectorized relationship coverage. The parameter-only Microsimulation could use the existing reference system if startup is not the intended contract; NIIT/CTC could release module caches at teardown. Measure isolated Linux RSS before those separate changes; this PR does not rewrite their tests.
Verify after the hub marks this draft ready (the PR workflow currently skips Python tests while draft), in this PR's existing Rest job: download
rest-test-timings, inspectrest-python-core-resources.txtmaximum RSS, then check everyrest-python-*-resources.txtagainst the approximately 10 GB target and compare JUnit file times, including the new groups. Routing alone does not establish memory safety. No moved files, full suites, or microsimulations run locally.Local validation:
make formatand three static guards (test_rest_python_groups.py,test_make_test_matches_ci.py,test_pr_runner_budget.py), one file at a time. All 21 checks passed: routing 12, Makefile/CI coverage 2, runner budget 7. Formatting and lint passed before each commit. Pytest discovery initially imported the full model; the rerun keeps the real package path but skips package initialization/conftests for these source/collection/command-only guards. The routing guard's own child collections use placeholders and import no model tests.Hub microsimulation impact
No model change: test routing only.
axiom: n/a (test routing, no rule change)