Skip to content

Raise the Full Suite - Rest timeout from 60 to 90 minutes - #9785

Merged
MaxGhenis merged 2 commits into
mainfrom
ci-rest-timeout-90
Oct 4, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
ci-rest-timeout-90

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Summary

Raises the Full Suite - Rest (Python + variables) job's timeout-minutes from 60 to 90.

Why

Passing Rest runs on 2026-10-04, measured from each head's check-run start and end times:

PR head Rest duration
#9769 (7b85bc1) 42.4 min
#9653 (6948513) 52.4 min
#9777 (a98d81d) 53.0 min
#9766 (b7b9cbe) 53.6 min
#9784 (13bb017) 54.6 min

With 5–18 minutes of headroom, a PR that adds a few minutes of tests gets cancelled at 60 minutes. #9695 (section 68 cap) was cancelled at the limit twice, at 11:16–12:21Z and earlier, although its own new tests pass in seconds. One earlier attempt also died to a runner shutdown at about 48 minutes; this change does not address that memory-pressure failure.

Change

  • .github/workflows/pr.yaml: Rest timeout-minutes: 60 → 90, with a comment recording the measurements.
  • Changelog fragment.

The durable fix is splitting "Run remaining Python tests" across two jobs. That is left as a follow-up.

axiom: n/a: CI configuration only.

🤖 Generated with Claude Code

Passing Full Suite - Rest runs took 42-55 minutes on 2026-10-04, so PRs
that add a few minutes of tests are cancelled at the 60-minute limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit d180905 into main Oct 4, 2026
35 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the US + core hub on the merge gates:

  • Head: a8edaea. That is the reviewed head 3c6203c plus a merge of main (picks up Run the Rest job's remaining Python tests in six pytest processes #9787). The PR diff is byte-identical.
  • CI: 35/35 check runs successful on this head. The split Rest job took 58 min (18:35–19:33), which shows the 60-minute limit was too tight.
  • Independent reviews (GPT-6.1 Sol): r1 APPROVE on 3c6203c; r2 main-merge APPROVE on a8edaea. Reports: ~/reviews/us-hub/lander/9785-review-r1.md and -r2.md.
  • Admin merge: used only for the required-review rule, at the reviewed head.

@MaxGhenis
MaxGhenis deleted the ci-rest-timeout-90 branch October 4, 2026 19:34
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
…ush job with a timeout (#10016)

* Skip, rather than fail, versioning for superseded merges

When several PRs merge before the versioning job gets a runner, every job
but the newest commits a version bump on a stale parent and fails on push.
Oct 6 00:00Z to Oct 8 16:28Z, 47 of 51 failed Push runs were exactly this:
in all 47, main had already moved past the job's commit before the job
started, and in every successful versioning run it had not. The newest
merge's bump consumes all pending changelog fragments (for example,
7c857cf removed the three fragments from the 13:02Z burst on Oct 8), so
the older jobs had nothing to do.

The job now reads main's head first. If main has moved past github.sha it
records a notice and skips the remaining steps, so the run succeeds and
"Push" failures on main again mean something. If the lookup fails, the job
proceeds as before. The job is not cancelled and keeps no concurrency group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound every Push job with a timeout from observed durations

Five Push jobs had no timeout-minutes and so inherited GitHub's 360-minute
default. Each now has a limit well above its Oct 6-8 maximum: ReleaseLock,
ReleaseLockTests and Lint 10 min (max 0.4, 1.4 and 0.7), versioning 20 (max
12) and Publish 20 (max 1.3). A ReleaseLockTests run that hung for 45 min
(37650158537) would have stopped at 10.

The push Rest job's limit rises from 60 to 90 minutes, matching the PR Rest
job after #9785. On push its p50 was 51 min and its successful max 58.8,
and run 37719726248 hit the 60-minute limit, which skipped that version's
Publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add changelog fragment for the Push versioning and timeout fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip superseded versioning only when main strictly contains the commit

The skip compared main's head SHA with this commit for equality, so any
mismatch skipped, including a stale read that returned an older head.
If the newest merge's job got such a read, every job would skip and no
version bump would land. The step now asks GitHub's compare API for the
status of this commit against main and skips only on "ahead" (this
commit is an ancestor of main and main has more). Identical, behind,
diverged, an empty answer or a failed lookup all run versioning as
before.

A new code-health test runs the step's script against a stub gh for
every compare status and for a failed lookup, and checks that every
later step in the job is gated on the result. Replacing the condition
with a looser one fails five of its nine cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant