Repository navigation
CI Test job takes 4-6 hours and has hit the 6-hour limit #2099
Description
Activity
Another instance: #2107's
Testjob (run 37091880974, attempt 1) ranRun tests with coveragefrom 05:01Z to 11:02Z on 2026-10-03 and was cancelled at the 6-hour limit. That PR only changes two property-test fixtures. Attempt 2 is running.🤖 Generated with Claude Code
- added a commit that references this issue
on Oct 3, 2026 Another instance, with timing: #2051's
Testjob (run 37021760212) started at 22:26Z on 2026-10-02. It was cancelled at the 6-hour limit at 04:26Z on 2026-10-03, after pytest had finished.policyengine-core test policyengine_uk/tests/policy: 1,633 passed in 2:18.pytest policyengine_uk/tests/: every test passed or xfailed. The last result (test_uc_unearned_benefits_properties.py::test_carer_support_payment_above_the_cap_does_not_change_uc XFAIL [100%]) is at 04:08:13Z, and there are no FAILED or ERROR lines. The remaining ~18 minutes went on whatever runs after the last test (coverage XML and teardown) until the cancel.
Per-file times from the gaps between
-vresult lines:File Minutes test_uc_unearned_benefits_properties.py 101.7 test_reform_representations.py 56.4 test_uc_state_pension_properties.py 43.9 test_pension_credit_reported_capital_properties.py 14.3 test_ctc_family_element_properties.py (new in #2051, max_examples=25)13.2 test_ni_class_4_properties.py 13.2 test_ni_threshold_freeze.py 11.4 test_housing_benefit_pension_age_properties.py 9.4 So a job whose tests all pass can still be cancelled. An explicit
timeout-minuteswould not help; the fixes listed above (a Hypothesis CI profile, xdist, a separate property job) would.🤖 Generated with Claude Code
- added a commit that references this issue
on Oct 3, 2026 Another data point: run 37092342044 for #2089 (branch
uc-hb-split-shared-cap-lha-rules).- The YAML stage passed: 1,782 tests in 4:13.
pytest policyengine_uk/tests/collected 2,341 items. 2,307 had passed, with 0 failures, when the job was cancelled at the 6-hour cap (06:19Z → 12:07Z). It had reached 98%, intest_uc_rebalancing.py.- The slowest files, from the gaps between PASSED timestamps on the macOS runner:
min file 53.3 test_uc_earnings_deductions_properties.py 48.9 test_reform_representations.py 48.1 test_uc_minimum_income_floor_properties.py 20.7 test_council_tax_reduction_pension_credit_properties.py 16.1 test_child_and_adult_definitions_properties.py 15.8 test_ni_class_4_properties.py 15.7 test_pension_credit_reported_capital_properties.py 11.8 test_benefit_cap_lha_scheme_split_properties.py (#2089's own) The Hypothesis property files dominate. A
ciprofile with a lowermax_examples, or a separate property-test job, would bring the Test job back under the cap.Status on 10 October 2026, from the UK hub:
- Raise the Test job timeout to 120 minutes #2258 raised the
Testlimit from 60 to 120 minutes as a stopgap (merged asd98af495). The job now runs onubuntu-24.04-arm. - Measured over the last 45 PR
Testruns (Actions API, about 20:30Z): seven successes took 58.5 to 59.6 minutes (median 58.7), and 18 runs were cancelled at 60.1 to 60.3 minutes. So the suite still takes about an hour, three times the 18 minutes the old comment recorded. - No PR addresses the runtime yet.
Two small corrections for whoever next edits the timeout comment or the changelog guidance:
- The comment added by Raise the Test job timeout to 120 minutes #2258 says every run on 10 October was cancelled at the limit; three passed (58.5, 58.7 and 59.6 minutes). Suggested wording: "In October 2026, successful runs took about an hour and multiple runs were cancelled near the previous 60-minute limit (issue PR Test job times out at 60 minutes: the suite now takes about an hour #2257)."
- Fragments that begin with
-render as a doubled bullet under towncrier's default template.
- Raise the Test job timeout to 120 minutes #2258 raised the
The
Testjob inpr_code_changes.yaml(macos-latest,make test) now runs close to GitHub's 6-hour job limit, and one PR's job has already hit it.What the logs show
pytest policyengine_uk/tests/All dates are 2026-10-02. The annotations also say "Due to capacity constraints, jobs targeting macOS arm64 runners may experience longer queue times". About 15
Code changesruns were queued for the macOS runner at 19:45Z.These are per-file pytest times, taken from the gaps between consecutive
-vresult lines:test_uc_unearned_benefits_properties.pytest_uc_state_pension_properties.pytest_uc_income_from_capital_properties.pytest_reform_representations.pytest_pension_credit_reported_capital_properties.pyMain has 12
*_properties.pyfiles at 3c48247, up from 5 at 143cc3c (#1896's head).test_uc_unearned_benefits_properties.pygrew by 342 lines on main over that range.Why it matters
Every PR into
mainnow waits several hours in the macOS queue and then about 4-5 hours inTest. The job spends about 24 minutes outside pytest (about 5 before, 19 after in run 37020241005), so a run whose pytest step goes past about 5h35m will be cancelled, whatever the code.Possible fixes (not yet tried)
pytest-xdiston the runner.Testtoubuntu-latestif nothing needs macOS.timeout-minutesto make the limit visible.Found while waiting on #2001's CI.
🤖 Generated with Claude Code