Skip to content

ci: run the test suite on four xdist workers - #469

Merged
hellices merged 2 commits into
mainfrom
ci/468-xdist-workers
Oct 23, 2026
Merged

hellices merged 2 commits into
mainfrom
ci/468-xdist-workers

Conversation

@hellices

@hellices hellices commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Closes #468

Summary

Every CI test job ran the whole suite in one process. On Python 3.12 the coverage job took 33-50 minutes: past its old 45-minute limit (#454), and within ten minutes of the new 60 on korvid-runners (#460's 3.12 took 49m58s). windows-test took 27-32 minutes. pytest-xdist was already a dev dependency that nothing ran.

  • Four workers: the Linux test matrix and the Windows full-suite step now pass -n 4.
    • GitHub-hosted runners for public repositories have four vCPUs.
    • A fixed count keeps a korvid-runners pod from claiming every core: it requests one CPU on a shared eight-vCPU node, and auto would start eight.
    • pytest-cov combines the workers' coverage, so the 80% gate is unchanged.
  • Unchanged Windows steps: the docs landing harness and the native-terminal smoke stay single-process, as before.
  • Deterministic test IDs: xdist needs every worker to collect the same test IDs. tests/k8s/test_pulse.py derived two IDs from gzip.compress() output, whose header carries the time it was written. Collections a second apart disagreed, and xdist aborted with "Different tests were collected between gw0 and gwN". Those four cases now have explicit IDs. Across two full collections three seconds apart, they were the only IDs that differed.
  • Pins and comments:
    • tests/test_ci_workflow.py pins the new commands.
    • tests/test_platforms.py updates its pin of the Windows command.
    • The quality-gates.md table shows the new command.
    • The 60-minute timeout comment now describes the single-process duration as history; the limit stays as headroom.

Verification

  • New pin: tests/test_ci_workflow.py::test_the_full_suite_runs_on_four_xdist_workers failed first on the current command and passes now. It requires the Linux suite command to be exactly uv run pytest -n 4 --cov --cov-fail-under=80, and the main Windows suite step to start with uv run pytest -n 4 -q.
  • Stable IDs: collecting tests/k8s/test_pulse.py twice two seconds apart now gives identical IDs: gzip-not-compressed, gzip-truncated, gzip-trailing-bytes, br-unsupported.
  • The CI command, locally: run with the tach plugin enabled as in CI:
How it runs Wall time Result
pytest --cov --cov-fail-under=80 (before, on 223fea6) 27m46s 13,694 passed, coverage 92.92%
pytest -n 4 --cov --cov-fail-under=80 (this branch) 8m27s 13,697 passed, 25 skipped, coverage 92.92%

The first run of the parallel command failed only tests/test_platforms.py's two pins of the old Windows command, which this branch now updates.

Run locally on macOS with Python 3.13 and the frozen lock. Each step ran directly with the project's environment rather than through make, because this machine cannot sync from PyPI:

Command Result
pytest -p no:tach tests/test_ci_workflow.py tests/test_platforms.py tests/k8s/test_pulse.py tests/test_docs_links.py tests/test_docs_build_config.py 134 passed
python scripts/check_source_size.py passed
ruff check src/ tests/ and ruff format --check src/ tests/ passed; 659 files already formatted
SKIP=mypy,source-size pre-commit run --files <changed files> passed
mypy no issues in 659 source files
tach check all modules validated

This PR's CI (run 39929214721) is the first Linux and Windows run of the parallel suite, on hosted runners because it changes .github/. Every required check passed. Job durations against recent hosted runs of the same jobs in one process (#456, #457):

Job One process Four workers (this PR)
test (3.11) 37m25s-38m11s 13m56s
test (3.12) 33m25s-42m58s 16m15s
test (3.13) 32m22s-36m53s 13m28s
windows-test 27m13s-31m34s 8m22s

The 3.12 job's log shows created: 4/4 workers, 13695 passed, 27 skipped in 916.20s, and Required test coverage of 80% reached. Total coverage: 92.76%, so the gate held across the workers.

Checklist

  • This closes an accepted issue in this repository that is assigned to me.
  • Tests cover the change, written to fail first where behavior changed.
  • make check passes locally.
  • User-facing docs and the README match the change, or nothing user-facing changed.
  • No security invariant is weakened: writes still need a keystroke approval
    and a fail-closed audit entry, and sensitive reads are still masked.

Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

No blocking code issues were identified, but successful Linux and Windows parallel-suite CI runs have not yet been established by the supplied evidence.

No open issues 🎉

What changed in this PR

This PR reduces CI test time by running the Linux and Windows full suites on four pytest-xdist workers, addressing #468 while preserving the 80% coverage gate.

Changes:

  • Enables four workers for full-suite runs; keeps the Windows docs and native-terminal steps single-process.
  • Adds explicit compression-test IDs to prevent timestamp-dependent collection mismatches.
  • Updates workflow assertions and documentation; retains existing timeouts.
File Description
tests/​test_platforms.py Updates Windows command assertions for four workers.
tests/​test_ci_workflow.py Pins worker flags and updates the timeout commentary.
tests/​k8s/​test_pulse.py Gives compression cases stable collection IDs.
docs/​dev/​quality-gates.md Documents parallel testing and combined coverage.
.github/​workflows/​ci.yml Enables four workers for Linux and Windows full suites.

Review effort: Balanced

@hellices
hellices requested a balanced review from Copilot October 23, 2026 09:23

@my-reviewer-agent my-reviewer-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

이슈 1건을 인라인 코멘트로 남겼습니다.

Comment thread tests/test_ci_workflow.py

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking issues remain, the existing coverage and Windows isolation controls are preserved, and the referenced CI run completed successfully.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved findings remain, and the referenced CI run completed successfully with parallel test execution.

No open issues 🎉

Review effort: Balanced

@my-reviewer-agent my-reviewer-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

검토 범위에서 보고할 이슈가 없습니다.

hellices and others added 2 commits October 24, 2026 00:12
Every CI test job ran the whole suite in one process. On Python 3.12 the
coverage job took 33-50 minutes, past its old 45-minute limit (#454) and
within ten minutes of the new 60 on korvid-runners. pytest-xdist was
already a dev dependency that nothing ran (#468).

The Linux test matrix and the Windows full-suite step now pass -n 4.
Hosted runners for public repositories have four vCPUs, and a fixed count
keeps a korvid-runners pod, which requests one CPU on a shared eight-vCPU
node, from claiming every core. pytest-cov combines the workers' data, so
the 80% gate is unchanged. Locally the exact CI command took 8m27s against
27m46s in one process, with the same 92.92% coverage.

xdist needs every worker to collect the same IDs. test_pulse.py derived
two from gzip.compress() output, whose header carries the time it was
written, so collections a second apart disagreed and xdist aborted. Those
cases are now named. test_ci_workflow.py and test_platforms.py pin the
new commands, and the timeout comment now describes the old
single-process duration as history.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Windows pin only checked that the command started with
`uv run pytest -n 4 -q`, so a command that later passed `-n 1` would
still pass it while xdist ran one worker, because the last `-n` wins.
_xdist_worker_counts now splits the command with shlex and returns every
worker count in each spelling pytest accepts (-n 4, -n4, --numprocesses
4, --numprocesses=4), and the pin requires exactly ["4"]. Six cases pin
the helper. Appending `-n 1` to the Windows command in ci.yml passed the
old pin and fails the new one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hellices
hellices force-pushed the ci/468-xdist-workers branch from 47194c3 to a19a7f5 Compare October 23, 2026 15:23

@my-reviewer-agent my-reviewer-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

검토 범위에서 보고할 이슈가 없습니다.

@hellices
hellices merged commit 9ab38db into main Oct 23, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: run the test suite on four xdist workers

2 participants