Repository navigation
PR Test job times out at 60 minutes: the suite now takes about an hour #2257
Description
Activity
Three more data points, from #2089:
- Head 9e9c461, run 38065669453, attempt 1 (job 114252679798, 15:57Z to 16:57Z): YAML 2,917 passed; pytest reached 99% with 3,520 of 3,523 items reported, 0 failed, then was cancelled at the 60-minute limit.
- Attempt 2 (job 114264742714, 16:58Z to 17:58Z): the same, with 3,519 of 3,523 reported and 0 failed.
- Head b432452 (main merged in), run 38078472542 (job 114290291765, 19:07Z to 20:08Z): YAML 2,917 passed; 3,503 of 3,526 reported, 0 failed, cancelled at the limit.
#2089 merges once a
Testrun can finish, which #2258 allows.Measured result: #2269 (shards) landed in rollup #2364 (merge f061b21) on 12 October.
- Before: one
Testjob ran the whole Python suite. Run 38075384588 took 59.6 min, and other runs were cancelled at the 60-minute limit (above). - After: on the rollup's CI run 38187059207 (head a8e4532, ten PRs in one run), the Python suite ran as three shards, all passing:
- shard 1: 25.1 min;
- shard 2: 33.7 min;
- shard 3: 24.5 min.
YAML took 4.2 min separately.
- These ran on shared runners at a busy time, so they're an upper bound rather than a benchmark.
Still open: #2306 moves to four shards and fails CI on any unexpected skip, xfail or XPASS. #2331 cuts each shard's peak memory.
- Before: one
What's wrong
The
Testjob inpr_code_changes.yaml(and the same job incode_changes.yaml) hastimeout-minutes: 60. Its comment says the job takes about 18 minutes. The suite now takes about an hour:9 October. Successful PR runs took 57 to 60 minutes (for example
fix-hb-income-exclusions13:06 to 14:04,legacy-award-readers-claimant-partner16:06 to 17:06).10 October. Every PR
Testrun since 11:04Z was cancelled at the 60-minute limit:fix-uc-protected-lcwra-2026fix-additional-dwelling-purchase-2238fix-uc-standard-allowance-uplift-2239fix-pension-relief-annual-allowance-2237axiom-parity-checkRun 38051896891 was cancelled at 99% with every test passing.
So no pull request can currently get a green
Test, and a rerun has the same odds.Fix
Raise the limit to 120 minutes in both workflows, so a run that is merely slow finishes. That is still well under GitHub's 6-hour cap, which the limit exists to avoid.
Separately, the suite's runtime has roughly tripled since the comment was written (the property-based tests added since are the likely cause). Bringing it back down needs its own piece of work.