Skip to content

Commit b994ce9

Browse files
Run the tiny e2e capstones on 3 runners instead of 9 (#1734)
* Combine tiny e2e legs into three runners Nine e2e matrix legs ran a few seconds of tests each on their own Depot runner, so ~80% of each leg was runner start, checkout, binary download and toolchain setup (#1732). Their toolchains don't overlap, so group them: - Poetry, PDM, Hatch and deno VEX capstones: one leg - Pipenv and pip VEX capstones: one leg (same uv + Pythons setup) - uv redirect, composer and .NET 10 capstones: one leg A row's test_filter may now hold several ';'-separated filters. The run step runs each suite once per filter in its own log group, and every suite x filter must still run at least one test. The tier test compares those expanded runs, so e2e-full rows still have to match a PR-tier run. Fixes #1732 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwAvjXtnjFQ1fNNf5f22eB * Guard blank e2e filter segments and skip a duplicate uv install A stray ';' in a combined test_filter produced a blank segment, which ran the suite binary with no arguments: every non-ignored test ran, passed, and the empty-run check could not tell. Fail the leg on a blank segment instead. The combined Poetry/PDM/Hatch/deno leg installed uv 0.11.19 twice: once in the PDM/Hatch step and again in 'Install pinned Poetry'. Skip the PDM/Hatch step when the leg also runs Poetry. Found by /code-review high on #1734. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012WGa6qSVVc5KBepGwGx3cJ * Skip blank e2e filters and tidy the matrix Review follow-ups for the combined e2e legs: - skip a blank `;` segment in test_filter instead of running the binary with no arguments, which selects the non-ignored tests, and make the tier test's runs() skip it the same way - check the PDM row through runs() instead of a second parser - put the PDM/Hatch, deno and .NET comments back next to the rows they describe - install uv 0.11.19 once for Poetry, PDM and Hatch instead of again inside the Poetry step Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwAvjXtnjFQ1fNNf5f22eB * Run Pipenv in the short VEX leg, not with pip The profiler's first runs of #1734 put the Pipenv + pip leg at 3.33-3.48 min, the last e2e job to finish in run 40411347506. pip alone is p50 2.88 min and was already the last job before ci-ok in 2 of 7 merge_group runs on 10-26, so adding Pipenv's ~0.5 min there lands on the merge-queue critical path. Move pipenv:: into the Poetry/PDM/Hatch/deno leg (~1.1 min) and give pip its own leg back. The runner count is unchanged (still 3 instead of 9). In the shared leg the Pipenv setup-uv (0.13.5) is first on PATH, so PDM/Hatch bootstrap with it instead of 0.11.19; that uv is tooling there, not the release under test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011f9aeP4Fqo6SktDCrA3ZuK * Give Pipenv its own e2e leg efa1296 moved pipenv:: into the Poetry/PDM/Hatch/deno leg. There the Pipenv setup-uv (0.13.5) goes first on PATH, and PDM 2.29 bootstrapped with it fails to lock: 'super(type, obj): obj (instance of PDMPyPIClient) is not an instance or subtype of type (Client)'. Both PDM capstones failed in run 40481294505. Pipenv can't share that leg, and with pip it would add ~0.5 min to the e2e critical path, so give it a leg of its own: 4 runners instead of the original 9, and the shared leg goes back to the set that passed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwAvjXtnjFQ1fNNf5f22eB * Revert "Give Pipenv its own e2e leg" This reverts commit 452247e. Its premise was wrong: the PDM failure on efa1296 was not caused by the Pipenv uv 0.13.5. On 452247e, with no setup-uv in the leg, PDM failed the same way. The cause is unearth 0.19.0 (2026-10-27) moving to httpx2, which breaks `pdm lock` on every PR; the next commit ports #1738's unearth<0.19 pin. Hatch passed under uv 0.13.5, so the 3-runner layout from efa1296 stands. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwAvjXtnjFQ1fNNf5f22eB * Port #1738's unearth<0.19 PDM pin unearth 0.19.0 (2026-10-27) moved to httpx2 and breaks `pdm lock` on PDM 2.x, failing the pdm e2e leg (and ci-ok) on every PR. Carry the same pin as #1738 so this PR can go green; it is a no-op once #1738 lands on main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011f9aeSWjd3xeW8riRfmswG --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7e9b27a commit b994ce9

2 files changed

Lines changed: 76 additions & 38 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 63 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1218,8 +1218,16 @@ jobs:
12181218
# matrix, 2.9.8 and macOS/Windows included, is
12191219
# composer-compatibility.yml.
12201220
# Both capstones share one leg per composer line: the setup is
1221-
# identical and the run step loops over `suite`.
1222-
- {os: ubuntu-latest, suite: e2e_vendor_composer_build e2e_redirect_composer_build, composer: '2'}
1221+
# identical and the run step loops over `suite`. The PR-tier leg
1222+
# also carries the newest-uv redirect capstone (described with the
1223+
# uv rows below) and the .NET 10 one: each suite runs in seconds,
1224+
# their toolchains (PHP, uv, .NET) don't overlap, and one runner
1225+
# instead of three saves ~0.6 job-min of fixed setup per suite per
1226+
# run (#1732).
1227+
# Real .NET SDK capstones: hosted + vendored nuget, one leg per SDK
1228+
# major (the suite pins the major through a sandbox global.json):
1229+
# the newest here, 7-9 on ubuntu in e2e-full.
1230+
- {os: ubuntu-latest, suite: e2e_vendor_composer_build e2e_redirect_composer_build e2e_redirect_uv_build e2e_nuget_dotnet_build, composer: '2', uv: '0.12.17', dotnet: '10'}
12231231
# Real-bundler gem capstones, one leg per bundler era. Boundaries:
12241232
# 1.17/2.1 merged GEM section, 2.2 separate sections, 2.5 last
12251233
# pre-CHECKSUMS, 2.6 CHECKSUMS, 4.0.15/4.0.21 before/after the
@@ -1347,32 +1355,39 @@ jobs:
13471355
# the oldest and newest line + the 0.5.x boundary (0.5.4 still
13481356
# re-resolves a transitive override / rejects a repointed
13491357
# constraint under --locked; 0.5.5 keeps both). The other 0.N
1350-
# lines and 0.5.3/0.5.6 are in e2e-full.
1351-
- {os: ubuntu-latest, suite: e2e_redirect_uv_build, uv: '0.12.17'}
1358+
# lines and 0.5.3/0.5.6 are in e2e-full. The redirect capstone
1359+
# rides the composer leg above.
13521360
- {os: ubuntu-latest, suite: e2e_vendor_pypi_build, uv: '0.12.17', test_filter: --include-ignored}
13531361
# The Poetry / PDM / Hatch / Pipenv / pip / deno manifest-less VEX
13541362
# capstones share ONE test binary (`e2e_vex_build`, a module per
13551363
# tool — one optimized link in test-release instead of six), so
13561364
# each leg's `test_filter` names its tool's module and keeps
1357-
# `--ignored`.
1365+
# `--ignored`. A `test_filter` may hold several `;`-separated
1366+
# filters: the run step runs the binary once per filter, and each
1367+
# one must run a test. The PR tier runs Poetry, PDM, Hatch, Pipenv
1368+
# and deno in one leg, since each tool's tests take seconds and a
1369+
# leg of its own spent ~80% of its time on runner start and setup
1370+
# (#1732). pip (~3 min of tests) keeps its own leg so it does not
1371+
# grow the e2e critical path. In the shared leg PDM / Hatch
1372+
# bootstrap with the Pipenv leg's uv (it is first on PATH).
13581373
# Real-Poetry hosted + vendored capstones (#[ignore]-gated,
13591374
# unix-only). One leg per major / lock format: 1.0 (lock 1.0),
13601375
# 1.1 (lock 1.1), 1.8 (lock 2.0), 2.0 (first lock 2.1 writer),
13611376
# current.
1362-
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'poetry:: --ignored', poetry: '2.4.3'}
13631377
# Real PDM / Hatch capstones (wiremock Socket API that also serves
13641378
# the hosted wheel; PyPI for the tool bootstrap + six).
13651379
# PDM: lock 2 (1.4), refused 3.1 (1.15) and 4.2 (2.7), 4.3 (2.8),
13661380
# the hishel<1 bootstrap window (2.25) and current.
1367-
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'pdm:: --ignored', pdm: '2.29.2'}
13681381
# Hatch: 1.0 (hatch.toml env vendoring refused, needs >= 1.2),
13691382
# 1.2, the virtualenv<21 window (1.9, 1.14) and current.
1370-
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'hatch:: --ignored', hatch: '1.18.1'}
1383+
# Real deno negative capstone (no hosted/vendored wiring exists for
1384+
# deno; VEX must attest nothing): the current major here, older
1385+
# majors in e2e-extended.
13711386
# Real Pipenv / pip capstones (mock patch server; the tools are
13721387
# bootstrapped from PyPI). `pipenv:` / `pip:` hold one or more
13731388
# space-separated releases the suite loops over. The middle
13741389
# Pipenv releases are in e2e-full.
1375-
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'pipenv:: --ignored', pipenv: '2026.8.0'}
1390+
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'poetry:: --ignored; pdm:: --ignored; hatch:: --ignored; pipenv:: --ignored; deno:: --ignored', poetry: '2.4.3', pdm: '2.29.2', hatch: '1.18.1', pipenv: '2026.8.0', deno: '2.9.7'}
13761391
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'pip:: --ignored', pip: '22 23 24 25 26'}
13771392
# Real-Maven hosted + vendored capstones, one leg per Maven line:
13781393
# 3.6 (pre http-blocker), 3.8 (resolver 1.6: no trusted checksums),
@@ -1407,13 +1422,6 @@ jobs:
14071422
# (socket-patch-vendor.sbt) capstones on the current 1.x line. The
14081423
# other sbt lines, JDK 21, the agent cells beyond coverage-docker's,
14091424
# Mill, scala-cli and macOS / Windows are sbt-compatibility.yml.
1410-
# Real .NET SDK capstones: hosted + vendored nuget, one leg per SDK
1411-
# major (the suite pins the major through a sandbox global.json):
1412-
# the oldest and newest here, 7-9 on ubuntu in e2e-full.
1413-
- {os: ubuntu-latest, suite: e2e_nuget_dotnet_build, dotnet: '10'}
1414-
# Real deno negative capstone (no hosted/vendored wiring exists for
1415-
# deno; VEX must attest nothing), one leg per major.
1416-
- {os: ubuntu-latest, suite: e2e_vex_build, test_filter: 'deno:: --ignored', deno: '2.9.7'}
14171425
runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS != 'true' && (matrix.os == 'ubuntu-latest' && 'depot-ubuntu-24.04-4' || matrix.os == 'ubuntu-22.04' && 'depot-ubuntu-22.04-4') || matrix.os }}
14181426
# The real-toolchain capstones loop several releases per leg (pip,
14191427
# pipenv) or bootstrap a tool from PyPI before the suite (poetry, pdm,
@@ -1505,9 +1513,10 @@ jobs:
15051513
uv --version
15061514
15071515
- name: Install uv (PDM / Hatch legs)
1508-
if: matrix.pdm != '' || matrix.hatch != ''
1516+
if: (matrix.pdm != '' || matrix.hatch != '') && matrix.poetry == ''
15091517
# The suites bootstrap the pinned PDM / Hatch into their own venv
1510-
# with uv; this uv is tooling, not the release under test.
1518+
# with uv; this uv is tooling, not the release under test. A leg that
1519+
# also runs Poetry gets the same uv from 'Install pinned Poetry'.
15111520
shell: bash
15121521
run: scripts/pip-install-retry.sh uv==0.11.19
15131522

@@ -1830,35 +1839,54 @@ jobs:
18301839
E2E_ALLOW_EMPTY: ${{ matrix.allow_empty }}
18311840
shell: bash
18321841
# Runs from the package root with CARGO_MANIFEST_DIR set, as
1833-
# `cargo test` would. `suite` may name several binaries; an
1834-
# `allow_empty` row skips the ones whose test file has not landed.
1835-
# Every suite a leg runs must run at least one test (per suite, so
1836-
# one suite's tests never hide another's empty filter): a renamed
1837-
# module, a dropped `#[ignore]` under the default `--ignored`
1838-
# selector or a stale `test_filter` would otherwise pass as 0/0.
1842+
# `cargo test` would. `suite` may name several binaries and
1843+
# `test_filter` several `;`-separated filters: every suite runs once
1844+
# per filter, each in its own log group. An `allow_empty` row skips
1845+
# the suites whose test file has not landed.
1846+
# Every suite x filter a leg runs must run at least one test (so one
1847+
# run's tests never hide another's empty filter): a renamed module, a
1848+
# dropped `#[ignore]` under the default `--ignored` selector or a
1849+
# stale `test_filter` would otherwise pass as 0/0.
18391850
run: |
18401851
set -uo pipefail
18411852
exe=''
18421853
if [ "$RUNNER_OS" = Windows ]; then exe=.exe; fi
18431854
cd crates/socket-patch-cli
18441855
export CARGO_MANIFEST_DIR="$PWD"
1856+
IFS=';' read -ra filters <<< "$E2E_TEST_FILTER"
18451857
status=0
1858+
n=0
18461859
for suite in $E2E_SUITE; do
18471860
if [ "$E2E_ALLOW_EMPTY" = true ] && [ ! -f "tests/$suite.rs" ] && [ ! -f "tests/$suite/main.rs" ]; then
18481861
echo "::notice::$suite has not landed yet; skipped (allow_empty)"
18491862
continue
18501863
fi
1851-
log="../../target/e2e-$suite.log"
1852-
# shellcheck disable=SC2086 # the filter is several libtest arguments
1853-
if ! "../../target/e2e-bin/$suite$exe" $E2E_TEST_FILTER 2>&1 | tee "$log"; then
1854-
status=1
1855-
continue
1856-
fi
1857-
passed=$(sed -n 's/^test result: .* \([0-9][0-9]*\) passed;.*/\1/p' "$log" | head -n 1)
1858-
if [ "${passed:-0}" = 0 ]; then
1859-
echo "::error::$suite ran no test in this leg (filter: $E2E_TEST_FILTER). Fix the row's suite or test_filter so it selects the tests it is meant to run."
1860-
status=1
1861-
fi
1864+
for filter in "${filters[@]}"; do
1865+
# A blank `;` segment would run the binary with no arguments
1866+
# (every non-ignored test), which the empty-run check below
1867+
# can't tell from a real selection.
1868+
if [ -z "${filter// /}" ]; then
1869+
echo "::error::$suite has an empty test_filter segment ($E2E_TEST_FILTER). Drop the stray ';'."
1870+
status=1
1871+
continue
1872+
fi
1873+
n=$((n + 1))
1874+
log="../../target/e2e-$suite-$n.log"
1875+
echo "::group::$suite $filter"
1876+
# shellcheck disable=SC2086 # the filter is several libtest arguments
1877+
if ! "../../target/e2e-bin/$suite$exe" $filter 2>&1 | tee "$log"; then
1878+
echo "::endgroup::"
1879+
echo "::error::$suite failed (filter: $filter)."
1880+
status=1
1881+
continue
1882+
fi
1883+
echo "::endgroup::"
1884+
passed=$(sed -n 's/^test result: .* \([0-9][0-9]*\) passed;.*/\1/p' "$log" | head -n 1)
1885+
if [ "${passed:-0}" = 0 ]; then
1886+
echo "::error::$suite ran no test in this leg (filter: $filter). Fix the row's suite or test_filter so it selects the tests it is meant to run."
1887+
status=1
1888+
fi
1889+
done
18621890
done
18631891
exit "$status"
18641892

‎scripts/tests/test_ci_e2e_tiers.py‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,14 @@ def rows(job):
3434
return rows_mod.job_rows(JOBS, job)
3535

3636

37+
def runs(row):
38+
"""The (suite, os, filter) runs the e2e run step makes of a row."""
39+
# A blank segment runs nothing (the run step fails the leg on one); a
40+
# row with no filter at all runs the job default.
41+
filters = [" ".join(f.split()) for f in row.get("test_filter", "").split(";") if f.strip()] or [""]
42+
return {(suite, row["os"], f) for suite in row["suite"].split() for f in filters}
43+
44+
3745
def job_text(job):
3846
return "\n".join(JOBS[job])
3947

@@ -50,10 +58,12 @@ def test_every_row_is_a_test_target(self):
5058
f"no test target {suite}")
5159

5260
def test_full_rows_only_add_releases_to_pr_suites(self):
53-
pr = {(r["suite"], r["os"], r.get("test_filter", "")) for r in rows("e2e")}
61+
# A row may bundle several suites and `;`-separated filters; compare
62+
# the (suite, os, filter) runs the run step expands them into.
63+
pr = {cell for r in rows("e2e") for cell in runs(r)}
5464
for row in rows("e2e-full"):
5565
with self.subTest(row=row):
56-
self.assertIn((row["suite"], row["os"], row.get("test_filter", "")), pr)
66+
self.assertLessEqual(runs(row), pr)
5767
pr_rows = [tuple(sorted(r.items())) for r in rows("e2e")]
5868
full_rows = [tuple(sorted(r.items())) for r in rows("e2e-full")]
5969
self.assertFalse(set(pr_rows) & set(full_rows), "a row in both tiers runs twice")
@@ -343,7 +353,7 @@ def test_excludes_exactly_the_cells_ci_runs_on_every_pr(self):
343353
self.assertEqual(len(excluded), len(ci))
344354
for row in rows("e2e"):
345355
if "pdm" in row:
346-
self.assertEqual(row.get("test_filter"), "pdm:: --ignored")
356+
self.assertIn(("e2e_vex_build", row["os"], "pdm:: --ignored"), runs(row))
347357
self.assertFalse(any("pdm" in r for r in rows("e2e-full")),
348358
"a pdm row off the PR tier would leave its cell unrun on PRs")
349359
versions = re.search(r"pdm: \[([^\]]*)\]", "\n".join(self.job)).group(1)

0 commit comments

Comments
 (0)