Repository navigation
CI: run the analyzer variant sweeps' rows in parallel, and skip the harness's rule-wide passes there #245
Description
Why
Since #172 (29 Oct), a full CI run takes 17-19 minutes instead of about 10. Median 17m07s and p90 19m16s over the 50 runs with a Windows leg between 29 and 31 Oct. The cause is one theory, DroppedTaskVariantTests.A_dropped_task_is_reported_in_every_form_and_a_kept_one_never: 1,837 rows in one class, so one xunit collection, run one after another. In the test results of runs 41180296321 and 41178154299, every other test of every project has finished by 334-453 s. After that, the theory runs alone on one core of a four-core runner. It takes 420-820 s depending on the runner's hardware, which is why the test step takes either about 7.5 or about 14 minutes.
Each row that expects a diagnostic also runs twice. CSharpAnalyzerTest runs the source as written, then again under #pragma warning disable for each expected id (SkipSuppressionCheck turns that off). Whether a pragma silences a rule is the same for every shape a row varies. The verifier's other pass, under an <auto-generated> header, never runs here: it is skipped for an analyzer that configures generated-code analysis, and every CMTK analyzer does (GeneratedCode.AnalysisFlags).
Measured locally on 31 Oct (Apple silicon, 16 cores, release/2.3.1 at the time, the theory alone through the in-process runner):
| Variant | Wall | CPU (user) |
|---|---|---|
| As it is | 186 s | 206 s |
-parallelMode all -maxThreads 4 |
111 s | 276 s |
-parallelMode all, default threads (16) |
131 s | 517 s |
| Pragma pass skipped, serial | 135 s | 150 s |
Pragma pass skipped, -parallelMode all -maxThreads 4 |
70 s | 191 s |
xunit.v3 4.0.2 (the pinned version) runs rows of one theory in parallel under "parallelMode": "all" in xunit.runner.json, with DisableParallelization to opt a class, method or row out. The [assembly: Parallelization(...)] attribute the xunit docs show is not in 4.0.2. Hand-partitioning the rows is not needed.
Proposal
parallelMode: allforCodoMetis.TypeKit.Analyzers.Tests, opting out any class that shares state. If the suite turns out flaky under it, split the theory into four classes by a stable key instead, with a test that the parts cover every row exactly once.TestBehaviors = SkipSuppressionCheckon the variant sweeps (DroppedTaskVariantTests,TypeArgumentConstraintKindTests,TypeArgumentConstraintVariantTests,QueryDelegateConditionTests). Keep the defaults on the focused tests of each rule, so they still run the pragma pass. Prove it by seeding a defect only that pass sees, and showing a focused test fails while the sweep passes (verify-the-guard).
Done when
Both changes are in. The analyzer test project's wall time on CI is measured before and after on both legs, in this issue. The theory no longer finishes after every other test project. The guard seed is shown caught. Expected, not yet measured on CI: a test step of about 6-7.5 minutes, bounded by EF on Linux (~386 s) or Conventions on Windows (~389 s).
Activity
CaffeinatedCoder commentedon Oct 31, 2026
Moved from 2.4.0 (#177) to 2.3.1 (#198), by the maintainer's decision on 31 Oct. 2.3.1's topic PRs are where the 17-19 minute runs are being paid right now, and this change is test code only: no package, no generated member, no changelog entry, so what 2.3.1 ships is unchanged. It reaches
release/2.4.0throughmain, as the 2.3.1 fixes do. The branch starts fromrelease/2.3.1.CaffeinatedCoder commentedon Oct 31, 2026
Implemented on
test/analyzer-sweeps-parallel-2-3-1(not pushed yet). Local measurement on 31 Oct, Apple silicon, 16 cores, the wholeCodoMetis.TypeKit.Analyzers.Testsproject throughdotnet test, two runs each:release/2.3.1(cc72fa4)3,947 tests pass on both. CPU goes up with 16 threads, which contend; a CI runner has 4, which is where the earlier per-theory measurement (4 threads: 186 s down to 70 s) applies. The CI before/after on both legs comes from the PR run.
The whole solution passes in the worktree (35,336 passed, 1,248 skipped: SQL Server query-variant rows, unchanged).
Guard seeds, each reverted by its inverse edit:
DroppedTaskVariantTestsfails 2 rows under the new mode.#pragma warning disable:IgnoredOutcomeAnalyzerTestsfails 16 tests in the pragma pass, andDroppedTaskVariantTestspasses. That pass is now the focused tests', as intended.CaffeinatedCoder commentedon Oct 31, 2026
CI on #250 (run 41187518150, one run, so one sample of the runner hardware):
On Windows the analyzer project is no longer the bottleneck. It finishes level with Conventions (278 s) and EF Core (261 s). On Linux, this runner was a slow one: EF Core took 494 s (386 s and 212 s in the before runs). The analyzer project (531 s) and EF Core now finish within 40 s of each other, sharing four cores. The theory's summed row time went up to 1,792 s from 763 s, because its rows now wait on each other for CPU.
What this leaves: on Linux the leg is now bound by CPU, not by one serial tail. The next cut is less work per row, not more parallelism: compile once per call, not once per row (unmeasured), and EF Core's two query sweeps of 12,275 rows each. #246 (Conventions) matters on Windows, where Conventions is now level with the analyzer project.
CaffeinatedCoder commentedon Oct 31, 2026
Done in #250, merged into
release/2.3.1as c8fb57c.Closesdoes not close an issue from a release branch, so this is closed by hand. The CI before/after is in the comment above.