Raise the Linux library build timeout to 24 hours - #2267
Open
mattgodbolt-molty wants to merge 1 commit into
Open
Raise the Linux library build timeout to 24 hours#2267mattgodbolt-molty wants to merge 1 commit into
mattgodbolt-molty wants to merge 1 commit into
Conversation
QuantLib 1.43 ran for 721 minutes against the 720-minute limit and was cancelled part-way through the clang/libc++ combinations, leaving the library unusable for those compilers until the run was resumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
|
Might just be better to only build these new libraries than to do a blanket change. There should be a runnable workflow for this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QuantLib 1.43 ran from 23:04 to 11:06, which is 721 minutes against
timeout-minutes: 720, so GitHub cancelled it. The conclusion readscancelledrather thanfailure, which makes it easy to misread as someone having stopped it by hand.It was cut off with 177 packages built. Resuming it (run 31000227299) took 2h18m rather than another 12 hours, because the builder skips combinations already uploaded or already recorded as failed, and took the count to 186.
To be clear about what the resume did and did not do, since an earlier version of this description got it wrong: it did not recover a set of missing clang/libc++ builds. Those combinations were attempted in the first run and failed, which
ce_install build-status list-failed --library quantlibconfirms as 85 recorded failures. QuantLib 1.43 genuinely does not build against libc++ on clang 14 and older, or on clang trunk; clang 15 upward is fine. That is a separate problem from this one.What the timeout costs us is the tail: any combination the sweep has not reached by minute 720 gets no attempt, and so does not even leave a failure record to distinguish "broken" from "never tried". That is the case for raising it.
For scale: SystemC finished comfortably inside the window at 2h16m for three versions. This is specific to libraries the size of QuantLib, around 977 translation units times the full matrix.
Side effects
The job runs on
lin-builder, a self-hosted runner, so the six-hour ceiling that applies to GitHub-hosted runners is not in play and 1440 is accepted.The one real consequence is worth stating plainly.
lin-lib-build.yamlis called viaworkflow_callfrom three scheduled workflows:scheduled-nightly-lin-lib-builds.yaml0 6 * * *scheduled-lin-lib-builds.yaml0 12 * * 1-6scheduled-lin-lib-builds-full.yaml0 12 * * 0Each fans out a matrix over every library with
max-parallel: 3, and none of these workflows sets aconcurrencygroup. At 720 minutes a job starting at 12:00 is guaranteed to be gone by midnight, before the next day's runs. At 1440 a single slow library can still be holding one of the three slots when the following day's 06:00 and 12:00 runs begin.In other words the current timeout is doing double duty: it bounds a single build, and it also happens to stop a daily sweep outliving its own cycle. This change keeps the first and gives up the second. If the overlap is unwelcome, the tidier fix is a
concurrencygroup on the scheduled workflows, or making the timeout an input so only manual runs of large libraries get the longer budget (adhoc-command-lin-builder.ymlalready does the latter withtimeout-minutes: ${{ fromJSON(inputs.timeout_minutes) }}).🤖 Generated with Claude Code