Skip to content

Raise the Linux library build timeout to 24 hours - #2267

Open
mattgodbolt-molty wants to merge 1 commit into
mainfrom
molty/lin-lib-build-timeout-1440
Open

Raise the Linux library build timeout to 24 hours#2267
mattgodbolt-molty wants to merge 1 commit into
mainfrom
molty/lin-lib-build-timeout-1440

Conversation

@mattgodbolt-molty

@mattgodbolt-molty mattgodbolt-molty commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

QuantLib 1.43 ran from 23:04 to 11:06, which is 721 minutes against timeout-minutes: 720, so GitHub cancelled it. The conclusion reads cancelled rather than failure, which makes it easy to misread as someone having stopped it by hand.

It was cut off with 177 packages built. Resuming it (run 31000227299) took 2h18m rather than another 12 hours, because the builder skips combinations already uploaded or already recorded as failed, and took the count to 186.

To be clear about what the resume did and did not do, since an earlier version of this description got it wrong: it did not recover a set of missing clang/libc++ builds. Those combinations were attempted in the first run and failed, which ce_install build-status list-failed --library quantlib confirms as 85 recorded failures. QuantLib 1.43 genuinely does not build against libc++ on clang 14 and older, or on clang trunk; clang 15 upward is fine. That is a separate problem from this one.

What the timeout costs us is the tail: any combination the sweep has not reached by minute 720 gets no attempt, and so does not even leave a failure record to distinguish "broken" from "never tried". That is the case for raising it.

For scale: SystemC finished comfortably inside the window at 2h16m for three versions. This is specific to libraries the size of QuantLib, around 977 translation units times the full matrix.

Side effects

The job runs on lin-builder, a self-hosted runner, so the six-hour ceiling that applies to GitHub-hosted runners is not in play and 1440 is accepted.

The one real consequence is worth stating plainly. lin-lib-build.yaml is called via workflow_call from three scheduled workflows:

Workflow Cron
scheduled-nightly-lin-lib-builds.yaml 0 6 * * *
scheduled-lin-lib-builds.yaml 0 12 * * 1-6
scheduled-lin-lib-builds-full.yaml 0 12 * * 0

Each fans out a matrix over every library with max-parallel: 3, and none of these workflows sets a concurrency group. At 720 minutes a job starting at 12:00 is guaranteed to be gone by midnight, before the next day's runs. At 1440 a single slow library can still be holding one of the three slots when the following day's 06:00 and 12:00 runs begin.

In other words the current timeout is doing double duty: it bounds a single build, and it also happens to stop a daily sweep outliving its own cycle. This change keeps the first and gives up the second. If the overlap is unwelcome, the tidier fix is a concurrency group on the scheduled workflows, or making the timeout an input so only manual runs of large libraries get the longer budget (adhoc-command-lin-builder.yml already does the latter with timeout-minutes: ${{ fromJSON(inputs.timeout_minutes) }}).

🤖 Generated with Claude Code

QuantLib 1.43 ran for 721 minutes against the 720-minute limit and was
cancelled part-way through the clang/libc++ combinations, leaving the library
unusable for those compilers until the run was resumed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@partouf

partouf commented Aug 5, 2026

Copy link
Copy Markdown
Member

Might just be better to only build these new libraries than to do a blanket change. There should be a runnable workflow for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants